This is the story of how I fell into the rabbit hole of proxy automatic configuration (or PAC) files. My company lets me connect to their network by VPN. But we also use cloud services like GitHub. The enterprise GitHub is secured by an IP allow list, so it can only be accessed from legitimate devices. In our case, that traffic comes from the outgoing forward proxy. The VPN somehow doesn't route all traffic through that proxy, so this needs to be configured on the client machine. So far so good, let's get started.

The simple solution

On macOS, you can go to Wi-Fi Settings > Your Wi-Fi name > Proxies. Let's set the HTTP and HTTPS proxy to "proxy.company.ch" and the port to 8080. Now I can access GitHub without any issues. I can work and everything is fine, until I want to google something. Wait, what? Google.com gives me an error. And so does pretty much the rest of the internet. I get an ERR_TUNNEL_CONNECTION_FAILED. It looks like the VPN can't resolve any other hostnames than github.com and a few others. Probably this was configured by hand, or there is no proper DNS resolver configured? I'll probably never know. If there were only a way to tell the system to use the proxy only for certain domains. Enter the world of PAC files.

The PAC file

Luckily, the company is hosting a PAC file. The Proxy Auto-Config file defines how web browsers and other user agents can automatically choose the appropriate proxy server for fetching a given URL. The file contains the JavaScript function FindProxyForUrl(url, host), which returns a string with one or more access methods.

Let's look at the file that my company provides:

function FindProxyForURL(url, host) {
  PROXY = "PROXY proxy.company.com:8080";

  if (shExpMatch(host, "*github.com|*.some-other-domain.com")) {
    return PROXY;
  }

  return "DIRECT";
}

Cool. Send traffic to github.com and other important domains through the proxy and everything else not.

Let's just use that file. Go to Wi-Fi Settings > Your Wi-Fi name > Proxies. Turn automatic proxy configuration on and set the URL to https://company.com/proxy.pac or whatever the link to the PAC file is. As soon as you click OK, the file is loaded from the server. However, in the browser, everything is like before. That's because the browser loads the PAC file on startup. So let's close the browser and open it again. Cool. GitHub.com works and so does the rest of the internet. All fixed, let's get to work. I use IntelliJ, and there is this neat feature to see GitHub pull requests in the IDE. But wait, it's not working. Let's go to the IntelliJ Settings > Proxy and activate "Auto-detect proxy settings." It still doesn't work. Maybe I also need to specify the PAC file location explicitly: click on "Automatic proxy configuration URL" and set the URL there. Maybe the IDE just needs a restart. Still nothing. What domains does the PR plugin connect to? I start a local intercepting proxy and point IntelliJ there, so I can analyze the traffic. It sends some POST requests to https://api.github.com/graphql.

Isn't that strange? api.github.com/graphql should definitely match *.github.com. Let's verify this online with a PAC file tester. The URL “http://api.github.com/graphql“ returns PROXY, and so does “http://github.com”. Let's try it also on this page for good measure. Same. How come IntelliJ is not sending this traffic to the proxy, but the browser is?

I fire up a transparent proxy so I can do a man-in-the-middle and analyze the requests and responses in detail: mitmproxy -p 9001 --mode upstream:http://company.proxy.com:8080. There are a couple of requests being made, all of them go to api.github.com .And guess what: The feature itself now works in IntelliJ. So it's not really the proxy behaving weird, and it must have something to do with the PAC handling in IntelliJ. So I create my own PAC file I can play around with.

What the PAC!

Let's try to find a PAC configuration that works for my use-case. There are rumors that you can use a local pac file, so I add file:///Users/myuser/proxy.pac in the WIFI settings and make a simple config to redirect all traffic to mitmproxy:

function FindProxyForURL(url, host) {
  return "PROXY localhost:9001";
}

That doesn't so anything. My experience is the same as these guys had here. And this discussion never got an answer but 150 up votes. Then I host my own PAC file on a server.

The hosted PAC

I use the same PAC file and start a webserver in that directory: python3 -m http.server 9001 --bind 127.0.0.1

Then I set the PAC config to the hosted file: http://localhost:9001/proxy.pac. Now that I have a solution, that actually works, I can play around with the config.

I immediately notice that there are a bunch of requests as soon as I click ok:

This makes sense because it needs to fetch the new config. But there is also a request as soon as I open Chrome or IntelliJ. Why is that? I thought that the network interface somehow routes the traffic to the correct location, but that's not how it works. When I specify a PAC file in the Wi-Fi settings AND configure a client application to use the system settings, then the client application gets, parses, and evaluates the PAC file itself.

A PAC file was invented by Netscape in 1996, so it's been around for a long, long time. It's just a JavaScript file with a bunch of helper or utility functions that are available. A PAC file contains the one function FindProxyForURL. Inside the function, helpers like shExpMatch and isInNet are available. For parsing a JS file, you need a JavaScript engine. Luckily, the browser already has a JS engine. It fetches the file, parses it, evaluates the code, and applies the rules for redirecting the traffic. But where do those helper functions come from? Well, they are baked into the browser. They are only available in a small sandbox called pac-sandbox inside the JS engine. This is also the place where the PAC file is executed.

We can see the implementation of the helper function shExpMatch in Chromium and in Spidermonkey

The implementation is the same:

function shExpMatch(url, pattern) {
  pattern = pattern.replace(/\./g, "\\.");
  pattern = pattern.replace(/\*/g, ".*");
  pattern = pattern.replace(/\?/g, ".");
  var newRe = new RegExp("^" + pattern + "$");
  return newRe.test(url);
}

We can see that this implementation can handle the pipe. Let verify this to be sure:

console.log(shExpMatch("http:test.example1.com", "*.example1.com|*.example2.com")) // ==> true

And what if a client doesn't bring its own JS engine already? Probably they use the pacparser library. This one too needs a JS engine and the library already bundles Spidermonkey. So maybe that's what IntelliJ does, maybe they wrote their own implementation.

Whichever is the case, the most likely reason that it doesn't work is that there is a bug in IntelliJ shExpMatch implementation. Let's see if this is the case.

The bug

The expression shExpMatch(host, "*.github.com|*.example.com) evaluates to true in Chrome, Firefox and Pacparser for host github.com and api.github.com. And it evaluates to false in IntelliJ. Finally, we found the bug Without the pipe | is all evaluates to true. It's the pipe that IntelliJ can't handle!

The shExpMatch Function is highly suspicious:

• findproxyforurl.com - "Will attempt to match hostname or URL to a specified shell expression"

• Microsoft - "The shExpMatch(str, shexp) function returns true if str matches the shexp using shell expression patterns."

• Mozilla "Currently, the patterns are shell expressions, not regular expressions."

What the hell is a shell expression anyway? What kind of matching does it do?

The fix

I need to circumvent the pipe. I could just write it like this:

if (shExpMatch(host, "*.github.com) || shExpMatch(host, "*.example.com)

But I want to stay away from shExpMatch for the rest of my life. So this seems a cleaner more stable solution:

function FindProxyForUrl(url, host) {
  PROXY = "PROXY company.proxy.com:8080";

  if  (dnsDomainIs(host, "github.com")) return PROXY;
  if  (dnsDomainIs(host, "example.com")) return PROXY;

  return "DIRECT";
}

The takeaways

Cool, I have a working solution and learned a ton along the way. My key takeaways are the following:

• If a pac file is specified in the WIFI settings, it doesn't route any traffic per se yet. It's merely a place where client applications can look up the URL to fetch, parse and evaluate the file.

• On macOS the pac file only works when it's hosted on a web-server. The local file with file:///path-to-file/proxy.pac has no effect. Now that we know that not the OS parses the file, but the client application, it appropriate to say that Chrome doesn’t support a local pac file. And indeed they removed the local pac file support: https://issues.chromium.org/issues/40574814

• In IntelliJ a pac file in UTF-8 only works if it doesn't use BOM. One more trap alone the way.

• In IntelliJ the shExpMatch Function breaks when there is a pipe (|) in the shell expression

A visual traceroute from Russia to Google’s public dns server 8.8.8.8

What is traceroute exactly and how does it work?

Traceroute is a network utility that sends out a bunch of requests (probes) and collects the responses of all network devices in between. But how come those devices send a response to the sender? Under normal circumstances, a device receives an ip packet, figures out where it needs to go and forwards it. Before doing so it decrements the TTL (time-to-live)value of the ip packet. The TTL value doesn’t really specify a “time” to live, it’s more a “number of hops to live”. A hop is the part of a network path between one network device and another. So if you send a TCP packet with a TTL value of 3, the first network device on the path (let’s say that’s your router at home) will decrease the TTL value to 2 and pass the package on to your ISP. The first device on the ISP’s network will decrease it further to a value of 1 and forward the packet again. The next device in the chain will reduce it again by 1 and the TTL value is now 0. A packet with a TTL value of 0 won’t be passed on anymore. Instead, the device will discard the package. From a sender’s point of view the packet has been dumped somewhere along the way, with no clue of when and where this happened. Luckily most devices are kind enough to let the sender know that they discarded a packet. They send a response using the ICMP protocol with a message that says: “Time-to-live exceeded (Time to live exceeded in transit)”. The default TTL values set by the operating system are high enough to ensure that packets can reach their destination before they die. But if we explicitly set a low TTL value we can trigger a response from devices between us and the destination. This is exactly what traceroute does. It sends out packets with a TTL of 1, 2, 3 and so on until the destination is reached. Each time the packet is able to travel one hop further and every time it expires along the way, we get a response.

Let’s see it in action

Fire up a packet sniffer like Wireshark(and set the filter to “udp and ip.addr == 8.8.8.8") or open the console and start tcpdump by typing “sudo tcpdump -v ‘icmp or udp’”. Go to another window in your console and start a traceroute to Google’s public dns server, which has the ip address 8.8.8.8 by executing “traceroute 8.8.8.8”. You will see an output similar to this:

A traceroute to the server 8.8.8.8

The number on the very left is the hop, followed by the ip address of the device and 3 response times in milliseconds. The idea is that those 3 times give you an idea of the average time. So it takes 9 hops from my PC to reach the server 8.8.8.8. For hops 7 and 8 I got a response from a different ip address, every time I sent a package. Let’s check the captured output of Wireshark to see what happened behind the scenes:

Wireshark capture of a traceroute to the server 8.8.8.8

We can see that three UDP packets have been sent to the destination 8.8.8.8. All of them have a TTL value of 1. Since they expired on the first hop, my router sent me three (one for each request) ICMP responses with the message “Time-to-live exceeded”. Then traceroute sent three probes with a TTL value of 2, but for these Wireshark never saw a response. Seems like my ISP is blocking ICMP responses on his gateway. A hop like this is called a black hole. Then traceroute sent UDP packets with a TTL of 3, 4, 5 and so on until it finally got an answer from the destination server (in our case the server with the ip 8.8.8.8). The server usually replies with an ICMP “Destination unreachable (Port unreachable)”. This is because the UDP packets are sent with a destination port number between 33434 and 33534, which is the reserved range for traceroute and not a valid port for an application. Pretty simple, but also very clever huh? Let’s see how we could build a tool like that in Node.js. What do we need?

1. We need to send UDP packets with a specified TTL.

2. We need to catch the ICMP responses and measure the elapsed time between request and response. We can use the destination port in the response to distinctively match it to a request and therefore increase the destination port for every request.

3. Traceroute can resolve a domain name to an ip address and also do a reverse dns lookup for ip addresses to get the symbolic name. So let’s also implement this.

4. Oh and let’s try to make everything with a maximum of 100 lines of code. Why? Because I’m sure we can!

A simple traceroute implementation in Node.js

First, let’s import some modules that we need. Dgram lets us send UDP packets and we use raw-socket to catch the ICMP responses. Dns-then is just a promise wrapper to the dns module that we use for dns lookup and reverse lookup.

Now we build a function that always sends out 3 packets, before increasing the TTL value and the port:

Next, we need to catch the ICMP responses, match the port number to the current request and do a reverse dns lookup if required.

The handleReply function called in the above snippet is in charge of formatting and logging the result. If the destination or the maximum number of hops is reached the program will exit. If not the sendPacket function is called again.

These are all the building blocks we need for the basic functionality of traceroute. Now we stick everything together and we are good to go. Check and download the complete source code.

Let’s run it and see what we get:

Output of our traceroute implementation

And let’s compare it to the native implementation:

Output of the built-in traceroute on OSX

As you can see the two traceroutes are very similar, but hops 7 and 8 are different. This is due to the fact that packets are being routed dynamically and don’t always take the same route, i.e. when you run the same traceroute twice you might get two different outputs.

Final Thoughts

This is only a very basic implementation to understand the core concept. The traceroute utility comes with a whole bunch of additional functionality like ASN lookup or support for different protocols. There are cases where firewalls block the UDP packets and you want to fall back to ECHO_PING for your requests. Play around with the code and implement additional features as you like. I hope you had fun and learned something.

Essential setup

Speed is essential. It is very important that if you can't write the commands on top of your head, you have a quick way of performing certain actions.

Take a minute or two to set up your environment properly before starting with the first question. Bookmark the Kubernetes cheat sheet and open the page as soon as the exam starts: https://kubernetes.io/docs/reference/kubectl/cheatsheet/. Copy and paste the following commands. Those three commands allow you to use the k alias for kubectl and have bash autocompletion activated for this alias. You can then type for example k get po a<tab> and it will autocomplete the pods name starting with a.

source <(kubectl completion bash)
alias k=kubectl
complete -F __start_kubectl k

I was reading a lot of articles where they suggested setting the following alias to quickly switch between different namespaces:

alias kn='kubectl config set-context --current --namespace'

I was always using this alias during the labs, but when the exam approached I realised that you sometimes need to save a command to a file. Since I was afraid the command couldn't just run as is, when I set the namespace permanently I decided to ditch the alias for the exam and always specify it explicitly.

Next set up your vim properly.

vim .vimrc

and add the following three lines:

set tabstop=2
set expandtab
set shiftwidth=2

With these set, you will be able to modify and add to the manifests without getting syntax errors because your YAML files are malformed.

Then set two more environment variables which will make your life much easier.

export now='--grace-period 0 --force'

You can use it to quickly delete a resource without waiting for Kubernetes for gracefully shut it down. For example: k delete po test $now. This will immediately terminate the pod and save you valuable time.

This next variable is maybe the most important one to set of all the tips above:

export do='--dry-run=client -o yaml'

You should use it anytime you want to create a manifest from the command line. So instead of doing

k run test-pod --image=nginx
k get po test-pod -o yaml > test-pod.yaml
k delete po test-pod $now

you can simply use the do variable to have a manifest:

k run test-pod --image=nginx $do

This will save you a huge amount of time.

Always use the command line and only check the docs if really needed

Try to use the command line to create resources whenever possible. You will be much faster using the CLI instead of searching the documentation. I think I pretty much completed the whole exam without using the documentation. Keep in mind that you can always use the -h flag to get useful examples.

Practise exams

I scheduled two practice exams from Killer Shell (https://killer.sh/) and found it very useful to practise for the real exam. The setup of the practice exams is very close to the real one and you can learn how you perform under time pressure. The practice exams are a bit harder than the real ones, so if you can get a good score in the practice exam you should be ready to rock the real one!

A Playstation 5 is pretty hard to get at the moment. I really wanted one and asked in a couple of stores. They all told me that they have a huge waiting list and won't even add me to it. When searching the internet I found a couple of pages where people would notify others when they saw an offer online. One for Switzerland is called https://www.preispirat.ch/. But whenever I saw a new post, it was already gone. So I decided to write my own small bot to notify me in Telegram was soon as one is available. I thought I need to wait for a week or so, but within a day I got multiple notification and was able to get a PS5. The setup is super simple and can be used for other shopping endeavours as well.

I checked https://www.mediamarkt.ch/ and saw that the have a handy text in the DOM, which I can use to determine when they are sold out. In case the text won't show up, I'll notify myself in Telegram.

The bot

The bot itself is a simple python script that requests the page and then checks if the sold-out text is not present. In that case it will send the message.

import telegram_send
import requests

url = "https://www.mediamarkt.ch/de/product/_sony-ps-playstation-5-digital-edition-980-pro-nvme-m-2-ssd-1tb-heatsink-2105983.html"

r = requests.get(url)

if "Produkt momentan nicht verfügbar" not in r.text:
    telegram_send.send(messages=[f"PS5 is available on {url}"])

Automation

I was running this on a small cloud server, but you can also run it on your own machine. First install telegram-send pip3 install telegram-send and requests pip3 install requests.

Run telegram-send --configure to configure telegram-send. It will walk you through on how to setup a telegram bot and connect to it.

Check if everything works fine and then setup a cron to run the script every 5 minutes. Edit your crontab:

$ crontab -e

and insert this line:

$ */5 * * * *  python3 /bytesonly/ps5-python.py

Then sit back and wait:

Good luck and happy shopping.

• It's called marshalling when you convert a xml object in memory to a serialized format like a string or file

• It's called unmarshalling when you convert a xml file or string to an object in memory (for exmpale to a java class)

With the @XmlRoot annotation

If the class you are trying to marshall has a @XmlRoot annotation it's pretty straight forward to marshall it.

val context: JAXBContext = JAXBContext.newInstance(Person::class.java)
val marshaller = context.createMarshaller()

// output pretty printed
marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true)

val sw = StringWriter()
marshaller.marshal(person, sw)
println(sw.toString())

Without a @XmlRoot annotation

I ran into a case, where I wanted to test a function that has @XmlType annotated java class as an input parameter. The verify function of the Mockk library always returned false, because the nested java object doesn't implement an equals function. Since this java class should be easily serialized I tried to marshall it using JAXB. I got an error saying that the class doesn't have an @XMLRoot annotation, which indeed was true. Since the class is from a third party library I had to find a way to marshall it with only a @XmlType annotation. Turns out this can be done by wrapping the object in a JAXBElement with a QName.

val jaxbElement = JAXBElement<Person>(
    QName("", "person"),
    Person::class.java,
    person
)

A full example would look like this. Let's say we have the following java class without a @XMLRoot annotation:

@XmlAccessorType(XMLAccessType.FIELD)
@XmlType(name = "person")
public class Person {
  String firstName;
  String lastName;

  Person(String firstName, String lastName) {
    this.firstName = firstName;
    this.lastName = lastName;
  }
}

Now we can marshall it using the following code:

val person = Person("firstName", "lastName")
val context: JAXBContext = JAXBContext.newInstance(Person::class.java)
val marshaller = context.createMarshaller()
val sw = StringWriter()
val jaxbElement = JAXBElement<Person>(QName("", "person"), Person::class.java, person)

marshaller.marshal(jaxbElement, sw)
println(sw.toString())