[Entry] Figuring Out HTTPS MITM in India

config.toml

@@ -9,7 +9,7 @@ pygmentsUseClasses = true
   # dir name of your blog content (default is `content/posts`)
   contentTypeName = "posts"
   dateFormat = "2006 Jan 2"
-  description = "asd"
+  description = "The Blog of Ceda EI"
 
   twitter = "https://twitter.com/ceda_ei"
   gitlab = "https://gitlab.com/ceda_ei/"

content/posts/figuring-out-https-mitm.md

@@ -0,0 +1,107 @@
++++
+title = "Figuring Out HTTPS MITM in India"
+date = "2021-12-25"
+author = "Ceda EI"
+tags = ["security", "internet"]
+keywords = ["security", "internet"]
+description = "Being served a MITM page over HTTPS"
+showFullContent = false
++++
+
+Belonging to India, I am very used to seeing random websites being blocked.
+However, today was particularly scary because the [MITM (Man In The Middle
+attack)](https://en.wikipedia.org/wiki/Man-in-the-middle\_attack) happened over
+HTTPS. I visited [usebottles.com](https://usebottles.com/) over HTTPS and was
+served with the following page.
+
+![](/images/usebottles\_censored.webp)
+
+Notice the padlock in the address bar. Checking into it, the certificate is
+valid and signed by [Cloudflare](https://cloudflare.com/).
+
+![](/images/usebottles\_certificate.webp)
+
+## Further Exploration and Hypothesis
+
+The initial hypothesis was that the Indian Government or the ISP has
+Cloudflare's signing keys and are serving the blocked page over HTTPS. This
+seems unlikely however and would be a very severe thing and would essentially
+erode all trust in HTTPS at scale as Cloudflare can sign any website's domain
+which essentially means that ISPs could MITM
+
+After a bit of exploration, the DNS entry of
+[usebottles.com](https://usebottles.com) points to `172.67.197.25`  and
+`104.21.92.184`. I checked that both of these IPs were owned by Cloudflare. To
+ensure that the DNS entries weren't being MITM attacked either, I checked for
+the same from my Hetzner Server.
+
+The second possibility that arises from this is that Cloudflare itself was
+serving the blocked page. While more likely than the previous scenario, it is
+still unlikely generally. I looked for any notices from Cloudflare about this
+and could not find any.
+
+At this point, I was mostly out of ideas. I looked into the source of the page
+and found something interesting. The entire page's source was the following
+(invalid) HTML:
+
+```html
+<meta name="viewport" content="width=device-width,initial-scale=1.0,maximum-scale=1.0" />
+<style>
+    body {
+        margin: 0px;
+        padding: 0px;
+    }
+
+    iframe {
+        width: 100%;
+        height: 100%
+    }
+</style>
+<iframe src="https://www.airtel.in/dot/" width="100%" height="100%" frameborder=0></iframe>
+```
+
+The most interesting part of this was that the iframe's URL pointed to
+[airtel.in](https://www.airtel.in/). Airtel is an ISP in India, however, I was
+not using internet services from Airtel.
+
+My presumption is based on this.
+
+## Final Hypothesis
+
+This is what I presume is happening.
+
+```
+Me <---> Cloudflare <---> usebottles' server
+     1                2
+```
+
+So far, we have been assuming the MITM is happening at `1` i.e. between Me and
+Cloudflare. However, the fact that `2` is secure isn't guaranteed.
+
+My best guess is that the Cloudflare server I am getting connected to happens
+to be using Airtel as the ISP. When Cloudflare's server tries to connect to
+usebottles' server, Cloudflare gets MITM attacked by their ISP - Airtel.
+Likely, SSL is not enforced between Cloudflare and usebottles' server. Thus,
+Cloudflare connects to usebottles' server over HTTP.
+
+Normally, a connection would happen the following way:
+
+1. I connect to https://usebottles.com/
+2. I get connected to Cloudflare's server.
+3. Cloudflare's server reaches out to usebottle's server.
+4. usebottles' server sends a response.
+5. Cloudflare signs the response with the certificate.
+6. I get a webpage over HTTPS.
+
+What seems to be happening is:
+
+1. I connect to https://usebottles.com/
+2. I get connected to Cloudflare's server.
+3. Cloudflare's server reaches out to usebottle's server.
+4. **Airtel intercepts the request and sends the blocking page**
+5. Cloudflare signs the **blocking page** with the certificate.
+6. I get **blocking page** served over HTTPS.
+
+I would be interested in knowing if there are any alternative explanations to
+this or something I have missed. You can [contact
+me](https://webionite.com/#contact) to let me know!

content/posts/python-poetry-pipx.md

@@ -0,0 +1,321 @@
++++
+title = "Improving Python Dependency Management With pipx and Poetry"
+date = "2021-09-19"
+author = "Ceda EI"
+tags = ["python", "development"]
+keywords = ["python", "development"]
+description = "My current dev setup with python, poetry and pipx"
+showFullContent = false
++++
+
+Over time, how I develop applications in python has changed noticeably. I will
+divide the topic into three sections and see how they tie into each other at
+the end.
+
+- Development
+- Packaging
+- Usage
+
+## Development
+
+Under development, the issues I will focus on are the following:
+
+- Dependency Management
+- Virtualenvs and managing them
+
+Historically, the way to do dependency management was through
+`requirements.txt`.  I found `requirements.txt` hard to manage. In that setup,
+adding a dependency and installing it was two steps:
+
+- Add the package `bar` to `requirements.txt`
+- Either do `pip install bar` or `pip install -r requirements.txt`
+
+While focused on development, I would often forget one or both of these steps.
+Also, the lack of a lock file was a small downside for me (could be a much
+larger downside for others).  The separation between `pip` and
+`requirements.txt` can also easily lead you to accidentally depend on packages
+installed on your system or in your virtualenv but not specified in your
+`requirements.txt`.
+
+Managing virtualenvs was also difficult. As a virtualenv and a project are not
+related, you need a directory structure. Otherwise, you can't tell which
+virtualenv is being used for which project. You can use the same virtualenvs
+for multiple projects, but that partially defeats the point of virtualenvs and
+makes `requirements.txt` more error-prone (higher chances of forgetting to add
+packages to it). The approach generally used is one of the following two:
+
+
+```
+foo/
+├── foo_src/
+└── foo_venv/
+```
+
+or
+
+```
+foo_src/
+└── venv/
+```
+
+I preferred the second one as the first one nests the source code one
+directory deeper.
+
+### A new standard - `pyproject.toml`
+
+In [PEP-518](https://www.python.org/dev/peps/pep-0518/), python standardized
+the `pyproject.toml` file which allows users to choose alternate build systems
+for package generation.
+
+One such project that provides an alternate build system is
+[Poetry](https://python-poetry.org/). Poetry hits the nail on the head and
+solves my major gripes with traditional tooling.
+
+### Poetry and virtualenvs
+
+Poetry manages the virtualenvs automatically and keeps track of which project
+uses which virtualenv automatically. Working on an existing project which uses
+poetry is as simple as this:
+
+```bash
+$ git clone https://gitlab.com/ceda_ei/verlauf
+$ poetry install
+```
+
+The `poetry install` command sets up the virtualenv, install all the required
+dependencies inside that, and sets up any commands accordingly (I will get to
+this soon).  To activate the virtualenv, simply run:
+
+```bash
+. "$(poetry env info --path)/bin/activate"
+```
+
+I wrap this in a small function which lets me toggle it quickly:
+
+```bash
+function poet() {
+	POET_MANUAL=1
+	if [[ -v VIRTUAL_ENV ]]; then
+		deactivate
+	else
+		. "$(poetry env info --path)/bin/activate"
+	fi
+}
+```
+
+Running `poet` activates the virtualenv if it is not active and deactivates it if
+it is active. To make things even easier, I automatically activate and
+deactivate the virtualenv as I enter and leave the project directory.  To do
+so, simply drop this in your `.bashrc`.
+
+```bash
+function find_in_parent() {
+	local path
+	IFS="/" read -ra path <<<"$PWD"
+	for ((i=${#path[@]}; i > 0; i--)); do
+		local current_path=""
+		for ((j=1; j<i; j++)); do
+			current_path="$current_path/${path[j]}"
+		done
+		if [[ -e "${current_path}/$1" ]]; then
+			echo "${current_path}/"
+			return
+		fi
+	done
+	return 1
+}
+
+function auto_poet() {
+	ret="$?"
+	if [[ -v POET_MANUAL ]]; then
+		return $ret
+	fi
+	if find_in_parent pyproject.toml &> /dev/null; then
+		if [[ ! -v VIRTUAL_ENV ]]; then
+		    if BASE="$(poetry env info --path)"; then
+			. "$BASE/bin/activate"
+			PS1=""
+		    else
+			POET_MANUAL=1
+		    fi
+		fi
+	elif [[ -v VIRTUAL_ENV ]]; then
+		deactivate
+	fi
+	return $ret
+}
+
+PROMPT_COMMAND="auto_poet;$PROMPT_COMMAND"
+```
+
+This ties in well with the `poet` function; if you use `poet` anytime in a bash
+session, activation switches from automatic to manual and changing directories
+no longer auto-toggles the virtualenv.
+
+![auto_poet and poet in action](/images/auto_poet.webp)
+
+### Poetry and dependency management
+
+Instead of using `requirements.txt`, poetry stores the dependencies inside
+`pyproject.toml`.  Poetry is more strict compared to `pip` in resolving
+versioning issues.  Dependencies and dev-dependencies are stored inside
+`tool.poetry.dependencies` and `tool.poetry.dev-dependencies` respectively.
+Here is an example of a `pyproject.toml` for a project I am working on.
+
+```toml
+[tool.poetry]
+name = "bells"
+version = "0.3.0"
+description = "Bells is a program for keeping track of sound recordings."
+authors = ["Ceda EI <ceda_ei@webionite.com>"]
+license = "GPL-3.0"
+readme = "README.md"
+homepage = "https://gitlab.com/ceda_ei/bells.git"
+repository = "https://gitlab.com/ceda_ei/bells.git"
+
+[tool.poetry.dependencies]
+python = ">=3.7,<3.11"
+click = "^8.0.1"
+questionary = "^1.10.0"
+sounddevice = "^0.4.2"
+SoundFile = "^0.10.3"
+numpy = "^1.21.2"
+
+[tool.poetry.dev-dependencies]
+
+[build-system]
+requires = ["poetry-core>=1.0.0"]
+build-backend = "poetry.core.masonry.api"
+
+# I will talk about this section soon
+[tool.poetry.scripts]
+bells = "bells.__main__:main"
+```
+
+One of the upsides of poetry is that you don't have to manage the dependencies
+in `pyproject.toml` file yourself. Poetry adds an `npm`-like interface for
+adding and removing dependencies.  To add a dependency to your project, simply
+run `poetry add bar` and it will add it to your `pyproject.toml` file and
+install it in the virtualenv as well. To remove a dependency, just run `poetry
+remove bar`. For development dependencies, just add the `--dev` flag to the
+commands.
+
+## Packaging
+
+Since poetry replaces the build system, we can now configure the build using
+poetry via `pyproject.toml`. Inside `pyproject.toml`, the `tool.poetry` section
+stores all the build info needed; `tool.poetry` contains the metadata,
+`tool.poetry.dependencies` contains the dependencies, `tool.poetry.source`
+contains private repository details (in case, you don't want to use PyPi).
+
+One of the options is `tool.poetry.scripts`. It contains scripts that the
+project exposes. This replaces `console_scripts` in `entry_points` of
+`setuptools`.
+
+For example,
+
+```toml
+[tool.poetry.scripts]
+foobar = "foo.bar:main"
+```
+
+This will add a script named `foobar` in your `PATH`. Running that is
+equivalent to running the following script
+
+```python
+from foo.bar import main
+
+if __name__ == "__main__":
+    main()
+```
+
+For further details, check the
+[reference](https://python-poetry.org/docs/pyproject/).
+
+Poetry also removes the need for manually doing editable installs (`pip install
+-e .`).  The package is automatically installed as editable when you run
+`poetry install`. Any scripts specified in `tool.poetry.scripts` are
+automatically available in your `PATH` when you activate the `venv`.[^1]
+
+To build the package, simply run `poetry build`. This will generate a wheel and
+a tarball in the dist folder.
+
+To publish the package to PyPi (or another repo), simply run `poetry publish`.
+You can combine the build and publish into one command with `poetry publish
+--build`.
+
+![example of poetry build](/images/poetry_build.webp)
+
+## Usage
+
+This part is more user-facing rather than dev-facing. If you want to use two
+packages globally that expose some scripts to the user, (e.g. `awscli`,
+`youtube-dl`, etc.) the general approach to do so is to run something like `pip
+install --user youtube-dl`. This install the package at the user level and
+exposes the script through `~/.local/bin/youtube-dl`. However, this installs
+all the packages at the same user level. Hypothetically, if you have two
+packages `foo` and `bar` which have conflicting dependencies, this causes an
+issue. If you run,
+
+```bash
+$ pip install foo
+$ pip install bar
+$ bar # works
+$ foo # breaks because of dependency mismatch
+```
+
+While installing `bar`, `pip` will install the dependencies for `bar` which
+will break `foo` after warning you[^2].
+
+To solve this, there is [`pipx`](https://github.com/pypa/pipx). Pipx installs
+each package in a separate virtualenv without requiring the user to activate
+said virtualenv before using the package.[^3]
+
+In the same scenario as before, doing the following works just fine.
+
+```bash
+$ pipx install foo
+$ pipx install bar
+$ bar # works
+$ foo # also works
+```
+
+In this scenario, both `bar` and `foo` are installed in separate virtualenvs so
+the dependency conflict doesn't matter.
+
+## Some more things from my bashrc
+
+```bash
+
+function wrapper_no_poet() {
+	local last_env
+	if [[ -v VIRTUAL_ENV ]]; then
+		last_env="$VIRTUAL_ENV"
+		deactivate
+	fi
+	"$@"
+	ret=$?
+	if [[ -v last_env ]]; then
+		. "$last_env/bin/activate"
+	fi
+	return $ret
+}
+
+alias wnp='wrapper_no_poet'
+alias pm='POET_MANUAL=1'
+```
+
+Prefixing any command with `wnp` runs it outside the virtualenv if a virtualenv
+is active. Running `pm` turns off automatic virtualenv activation.
+
+
+[^1]: This also allows for a nice switch between the development and production
+    versions of the app. Essentially, when the virtualenv is active, you are
+    using the development script while when it is deactivated, you are using
+    the global (likely production) version.
+
+[^2]: To be precise, it will warn you that it broke `foo` but will still
+    continue with the installation
+
+[^3]: For development, poetry also provides `poetry run` which runs a file
+    without having to activate the virtualenv.

themes/codex/layouts/_default/baseof.html

@@ -36,7 +36,7 @@
   {{ end }}
 
   {{ range .AlternativeOutputFormats }}
-  {{ printf `<link rel="%s" type="%s+%s" href="%s" title="%s" />` .Rel .MediaType.Type .MediaType.Suffix .Permalink $.Site.Title | safeHTML }} 
+  {{ printf `<link rel="%s" type="%s+%s" href="%s" title="%s" />` .Rel .MediaType.Type .MediaType.FirstSuffix.Suffix .Permalink $.Site.Title | safeHTML }}
   {{ end }}
   {{ block "links" . }} {{ end }}
   {{ partial "seo-schema.html" .}}