I work with notebooks a lot, especially when it comes to prototyping. 

The term "notebook" represents either a jupyter notebook or a jupyter lab notebook.

Whenever I discover interesting things in notebooks, I will publish them here.

{{ toc }}

### Run shell commands from cells

You can run shell commands (`ls`, `pwd`, `env`, `ssh`, etc.) from your notebook cell. First type a bang, then type the shell command.

```
!env | grep CONDA
```

### Assign output of shell commands to a variable

```python
out = !ls /Users/davidkuda/.ipython/profile_default/startup

for i in out:
    print(i)
```

### Load things from a python src file

If you happen to always do the same things at the beginning of each notebook or a session, you could simplify it by defining all these things in a python src file, and then run that file.

Let's say that by default, you always have the same imports:

```python
import json

import numpy as np
import pandas as pd
import seaborn as sns
```

Save the same lines in `namespace.py`, and make them available in your notebook session:

```python
%run namespace.py
```

Make sure that you get the path to the folder right, e.g.

```python
%run /Users/user/code/project/notebooks/namespace.pu
```

### Load things on startup

Create any `.py` file in:

```sh
cd $HOME/.ipython/profile_default/startup
```

As long as all the scripts can be executed without raising an exception, your namespace will be updated with whatever you define there.

Example:

```sh
echo 'NAME = "DAVID"' >> $HOME/.ipython/profile_default/startup/consts.py
```

Now, from any ipython session (i.e. in every notebook), the variable NAME is available to you.

### Use jq to format json

It's common to work with JSON. Sometimes even with complex json.

`jq` is a fantastic tool to add syntax highlight and nice formatting to your json strings.

You can save a json string as a variable, for example the return value of an API. You can then pass this value to jq directly from your notebook.

```python
s = '{"hello":"world"}'
!echo '{s}' | jq
```

You could even wrap this into a python function and invoke the python function instead. That way, you save a few characters to type.

```python
def jqit(t: str, filter: str = "."):
    !echo '{t}' | jq '{filter}'

jqit('{"hello":"world"}')
```

Result:

```json
{
  "hello": "world"
}
```

For such a small json string, this is of course over-engineered. 

But I often work with APIs that return huge json strings, especially when I work with APIs from NoSQL applications.

```python
import requests

def jqit(res: requests.models.Response, filter: str = "."):
    t = res.text
    !echo '{t}' | jq '{filter}'
```

Jupyter notebooks is fantastic for prototyping. Here are tips that will help you be effective with it.

Effective Jupyter Notebooks

Recently, I was talking with someone who told me: “Oh, you have no production experience with technology x, but only on your side projects? That’s not worth much!”

In fact, I would even say I learn more from side projects than from work projects.

At work, other people have laid out a path that you can follow. Patterns already exist. At work, you have to understand existing abstractions, and you have to know how to use existing patterns.

In side projects, you have to figure out everything yourself. And you have to take care of everything yourself.

By writing a command line application in Python, I learned how to run Python scripts in a shell with arguments, and I am solving repetetive tasks this way.

By programming my blog, I learned how web technologies (frontends and backends) are deployed and work together.

## Python Password Manager
This wasy my first meaningful, individual side project. I started it in 2020. At that point, I was comfortable with Python. At work, I haven’t designed a Command Line Interface (CLI, a terminal application), but I have used many.

I love CLI apps with a great user interface, I think it’s a great place to interact with your computer. So I wanted to have the ability to build such applications myself. 

At work, you can’t just do your own thing and play around with the code. Normally, you have to prioritise heavily and focus on what the business wants you to focus on. And of course, you also have to align your efforts with your team. You can’t spend much time on things that yield little value.

In your side projects, you can do whatever you want. You want to learn about cryptography and encode your data? Go ahead and do it! You don’t like the current design and APIs, and want you to change it? No one will ever stop you.

In your side projects, you can always focus on learning and growth. At work, learning and growth happen, too, but you have to focus on productivity.

Although there are plenty password managers out there, I wanted to program my own. There was just so many things that I could learn: How do you store sensitive data securely? How can you make sure no one can access them? How do you interact with your data structures? How can I make sure that it’s fun to use?

So I went ahead and read the official Python documentation of the `argparse` module and put everything into practice. I learned so much by doing that.

In the mean time, I read books like Clean Code or A Philosophy of Software Design, and I experimented with my code and APIs every day. I learned how to hide complexity and information, I learned how to avoid pass through methods, and much more.

It’s easy to write a script and run it in your IDE. But how can you execute your scripts from anywhere in your shell? You make your script executable by defining the “hashbang” (for example `/usr/bin/env python`)  at the top, by changing the file permissions with `chmod 744 script.py` and finally by putting it in a directory that is listed in the `$PATH` envrionment variable.

Nowadays, these skills are very useful at work. But at work, I had to focus on the business logic, and much less on obsessing with the user interface, code design and the shell environment.

## Building My Blog kuda.ai
- How react works, state patterns
- Server Side Rendering, Static Sites
- What a “Headless CMS” is

## Building lyricsapi
- writing a backend in golang
- RESTful APIs
- OpenAPI
- Authentication
- SSL


Recently, someone told me that my side projects are not worth much. But he didn't know: I had a ton of fun in my side projects, and I acquired many skills in my side projects.

Skills You Acquire In Side Projects

I asked ChatGPT what the most common psql commands, and found the answer so convincing that I wanted to store them on my blog.

{{ toc }}

![chatgpt-psql](//images.ctfassets.net/pedj0c0bs6fa/455lVOD9FygP6n6qdseVWH/c2dfa46b3602f8f12a803e66569cebe9/Screenshot_2023-01-29_at_20.30.02.png)

### Q: what are the most common psql commands?

ChatGPT:

```text
SELECT: retrieves data from one or more tables
INSERT: inserts data into a table
UPDATE: updates existing data in a table
DELETE: deletes data from a table
ALTER: modifies the structure of a table
CREATE: creates a new database, table, index, etc.
DROP: deletes an entire database, table, index, etc.
GRANT: assigns privileges to a user
REVOKE: removes privileges from a user
COMMIT: saves changes made during a transaction
ROLLBACK: undoes changes made during a transaction.
```

### Q: what are the most common psql options, for instance when you want to list all tables?

ChatGPT:

```text
-U or --username: specifies the PostgreSQL username to connect as
-d or --dbname: specifies the name of the database to connect to
-h or --host: specifies the database server's host name or IP address
-p or --port: specifies the database server's port number
-W or --password: prompts for a password before connecting to the database
-l or --list: lists all available databases on the server
-c or --command: executes a single SQL command and quits
-f or --file: executes commands from a file and quits
-q or --quiet: doesn't display any message
-v or --set: sets psql variables, for example: -v table=mytable (\l+ in psql will show all tables including mytable).
```

Example:

```css
psql -U [username] -d [database_name] -c "\dt"
```

### Q: what are the most common backslash commands in psql?

ChatGPT:

```text
\l or \list: lists all available databases on the server
\c or \connect: connects to a database
\dt or \dT: displays tables in the current database
\d [table_name]: displays the structure of a table
\du or \dU: displays the list of users and their roles
\q or \quit: quits psql
? or \help: displays help on backslash commands
\x or \x auto: toggles expanded table output mode
\g or \g [file]: executes the current query and optionally saves the result to a file
\i [file]: executes commands from a file.
```

I asked ChatGPT what the most common psql commands, and found the answer so convincing that I wanted to store them on my blog.

The Most Common psql Commands according to ChatGPT


Programming feels like a superpower. Whatever you can think of, you can build. Your mind is the only limitation.

Often I see products that are still missing in this world. I hope I will have time one day to build these products.

I will present my ideas here. Is it stupid to share product ideas? Many startup experts will tell you that the idea is no worth anything. It's the execution that's worth everything.

{{ toc }}

### CoLinkSpace: A Multi Player Bookmark Manager

I hoard knowledge: Books, videos, articles, newsletters, magazines, and links. Many links.

There are a myriad of ways how you can organize your links.

#### Browsers

A common way is using the bookmarks manager of your browser. This comes with problems, though. Your bookmarks are bound to your browser. Sure, you can use Firefox on your iPhone and on your Macbook and share the bookmarks that way. But what if you use many browsers? I do: Brave on my Macbook, Safari on my iPhone (because it enables dark mode via an extension), and Chrome for Work.

I have used Firefox for a long time. Although I could export my bookmarks, I didn't do it.

#### Bookmark Apps

Bookmark apps like InstaPaper or Pocket help you organize bookmarks across different browsers. They even provide additional useful features. They allow you to save websites, so that you can read them offline. Their UI optimise for reading. There is no noise, no ads. 

Great stuff, but these tools are single player only.

#### WhatsApp, E-Mails And Other Chats

My wife and I send us many links back and forth. She has recently send me an interesting article on parenting that appeared in nzz.ch.

I have sent this article to a couple of my peers, too.

So I have many useful articles persisted in the chat with my wife. I often search links there.

Then, I also send links via email. Sometimes I have to search in my email and on WhatsApp. And sometimes I end up not finding the article. 

WhatsApp enables you to create group chats. You could create a chat with your colleagues and share links related to work. But usually, you will also send many text messages back and forth. And the only way you can organize your links is by time. The latest shared link will always be the first.

#### Note Taking Apps: The Jacks of All Trades

I store my links in Bear, a beautiful note taking app for Mac. I can access the list from my Mac and from my iPhone. I open bear and search the file that holds my links.

This file contains multiple categories represented as titles.

To every link, I add a little description.

To find a link, I search either in the file, or globally.

This works really well. Though Bear is only single player, there are plenty of note taking apps that support collaboration. An outstandingly prominent candidate is Notion.

So, what are the disadvantages?

Well, note taking apps are so generic, they can do pretty much anything. And that's the problem: It has no focused scope on managing links. Writes are awkward: Open the app, search the file with the links, scroll through the page until you find a category that fits your link, append it, and add some information to it. What if a link could fit multiple categories? You could duplicate the link, but that's hard to maintain.

The reads are also not ideal. Usually, I use the global search or local search to find a link. This doesn't work too bad. But I can't search for only the website, for tags, for categories, for people I shared it with, for the title, for the description, and so on. The search is effective, but limited.

Conclusion: Note taking apps can do everything, but not necessarily with a great user experience. And I want a great user experience, that's why I use Bear, and not Apple's notes app.

#### CoLinkSpace: A Multi Player Bookmark Manager

In CoLinkSpace, users can create spaces and add links to these, cooperatively. 

Let's make an example. I open a space called "parenting". My wife joins that space. Whenever we find an interesting link, we add it to our space. 

We may invite other curators, if we want. If we think our list is useful to all parents, we may make our space public. So a space can be private and accessed only by members, or it can be public and accessed by anyone in the internet. 

Every link has a preview. You can add tags, if you want. That way, you can filter by tags. You can add a description. You can let other members know, why a link is worth reading, or what you liked most about it. Whenever a member of a spaces adds a link, the others get a notification. 

### WishList: Receive only things you want, not waste.

I don't like the name yet for this product, but I love the idea.

I am sure you are familiar with such situations: Your wife's mom will turn 60. What are you going to give her as a birthday present? Christmas is coming. What am I going to give my mom? My brother marries. What present will they enjoy and use?

In WishList, I can add all the things that I would like to have. Maybe I want to have noise canceling headphones from Bose. I can add it to my list. 

My friends and family can join my wish list. They can organize their presents on my WishList-site. A present may be expensive. Many people can share the price, and make the gift together. 

It's a tragedy that we buy so many things that we don't use. Every item gets manufactured. Every item needs natural resources. And so many items will never be used.

WishList will make your friends and family happier, because they will only receive things that they truly want and enjoy. WishList will make the economy more sustainable, because we will reduce waste. 

### Further Ideas:

- OrgDashboard: An intranet for companies and organisations with dashboards and other content to foster effective internal communication.
- Random Chords Training for musical instruments, e.g. guitar
- terminal app similar to jupyter notebooks. Feels like editing a markdown file, avoids the clumsy data structure on disk with the json (i.e. lightweight), and is open to all programming environments, natively.
- InteractiveJQ: JQ is a great tool to manipulate json files in your terminal. There are already great tools that enable you to interactively run jq queries, and I would like to build something even better.
- Open source booking platform for non profit orgs, e.g. for retreat centers to organize food and beverages and bed and breakfast etc.
- clipboard stash: sometimes we copy and paste the same line of text over and over again. why not assign keyboard shortcuts to a clipboard?
- Content Calendar: Plan your content, and have it deployed automatically
- Personal Kanban: Something in between Notion and Trello, but optimised for todo planning

Product Ideas

I often need to look up commands and their respective options. Normally, I would google my question. Let me give three recent examples from this morning: 

- _"How to set path with wget"_
- _"What is the d option in unzip?"_
- _"adduser gecos"_

Eventually I will find a post on stackoverflow or another stackexchange website that delivers good answers. [One of these answers](https://askubuntu.com/questions/420784/what-do-the-disabled-login-and-gecos-options-of-adduser-command-stand) has changed my behaviour. I knew about the man pages, but I have found it always rather cumbersome to skim through the man pages. I now learned how it can actually be easy to navigate the man pages.

As "Radu Rădeanu" answered in 2014, type `/<word>` to highlight every "word" in the man page, and press `n` to go to the next occurence. That's a really nice feature of the man page, and I think it's worth investing more time into getting man page behaviours straight. Thank you, Radu!

- `/word` -> search "word" after the cursor
- `?word` -> search "word" before the cursor
- `n` -> Go to next occurence of word
- `N` -> Go to previous occurence of word
- `d` -> Go downwards half a page
- `u` -> Go upwards half a page

If you search an option like `-s`, you will find it really difficult to find the description of your command. You can try it in `man curl`: You will find way too many highlights and be ineffective in your search.

In that case, you can use a regex / regular expression:

```bash
# search for -s; . -> any number; * -> any character; --
/-s.*--
```

```bash
man cp # open the manual page for the cp command
/-d # search "-d" inside the man page
n # go to the next occurence of "-d"
n # ...
N # go to previous occurence of "-d"
?-r # Look for occurences of "-r" before the cursor
```

Navigate the man pages

![code-top-to-bottom](//images.ctfassets.net/pedj0c0bs6fa/61bO3lIhnYfJQGG1NG9I5a/8908b1da7aed4e8a99de9bd19d5cfe49/Screenshot_2023-01-25_at_14.30.00.png)

When you read a document, you read it from top to bottom. 

You start by reading a summary, and you get a feel for what is inside the document from reading the table of contents. 

Then you naturally expect the content to be listed according to the table of content. 

Why not follow the same pattern for your code?

Your main function can be the same as a table of content. It can list the content of your code in a linear way and help to summarise what is happening. 

So a very simple way to make your code easier to understand is to put your main at the top, and then define all functions according to the order they are called in the main function. And define your functions according to that order.

Let’s see the following example to make the point clear:

```python
def main():
    """This app gets metadata from links and persists them"""
    html_text = get_html("www.kuda.ai")
    meta_data = parse_meta_data(html_text)
    persist(meta_data)

def get_html(url: string) -> HTMLText:
    ...

def parse_meta_data(content: HTMLText) -> MetaData:
    ...

def persist(data: MetaData):
   ...

if __name__ == "__main__":
    main()
```


Make sure that your code reads from top to bottom, without too many jumps. This will make your code easier to understand.

Make Sure Your Code Reads From Top to Bottom

![shell_script_functions](//images.ctfassets.net/pedj0c0bs6fa/5DCdIV5NDOmGTUGi6aGirO/a96dae749effa28169fb2faff9a856f3/shell_scripts.png)

Many applications do not require coding. Many problems can be solved with unix programs and pipelines. Python is good for glue, and so are shell scripts. In fact, shell scripts may even be simpler.

In the past few years, I have used shell scripts extensively, either for my own tools, for CI/CD pipelines, or for my applications. In this post, I want to store that knowledge, and I hope it will be useful for other computer programmers and engineers.

{{ toc }}

## The HashBang
This is a sample hash-bang: `#!/bin/zsh` (or more commonly: `#!/bin/bash` or `#!/bin/env bash`). You start an executable script with a hash-bang. The hash-bang defines the environment or application that will execute your code. You put it at the top of the script. You may put comments above the hash-bang. 
```sh
# script.sh
# Author: David Kuda
# Date: January 2023
#!/bin/zsh
echo "Hello World"
```
_This script will be executed in the environment defined in the hash-bang. This means that `zsh` will execute `echo “hello world”`_

The hash-bang is also known as “sh-bang”.

You can also execute python files, even from virtual environments or from conda:

```python
# greet.py
#!/usr/local/Caskroom/miniconda/base/bin/python3
import sys

name = sys.argv[1]
print(f"hello {name}")
```

```sh
# to execute the python script:
chmod 700 greet.py
./greet.py David
# prints: hello David
```

## chmod
Once you have a file script that you would like to execute (say with a double click, or in your terminal with `./script.sh`, you first need to change the file permissions. `chmod` changes these file permissions. 

```sh
# change file permissions:
chmod {permissions} {file_path}

# examples:
chmod 744 script.sh
chmod 400 sshkey
```

There are three entities in a unix environment:

- First Digit: You, the user
- Second Digit: The group, you belong to
- Third Digit: All others

chmod takes three digits. The first digit is for the user, the second for the group, and the third for all others.

```sh
chmod 744 script.sh
# 7 for the user: allow read, write, and execute
# 4 for the group and all others: allow read-only

chmod 400 sshkey
# 4 for the user: allow read-only
# 0 for the group and others: allow nothing (no read,
#     no write, no execute)
```

Many tutorials or stackoverflow posts will advise to use `chmod +x {file}`, but I think that’s a bad idea. `+x` translates to `777`, which means that you, your group and everyone else gets the permission to execute your file. `744` or `700` is safer!

- 0: none
- 1: execute only
- 2: write only
- 3: write and execute
- 4: read only
- 5: read and execute
- 6: read and write
- 7: read, write and execute

## The First Column of `ls -l`
`ls -l` prints the permissions of all files and directories:
```
> ls -l
drwxr-xr-x david staff 192 B  Tue Jan 24 13:43:39 2023  blog
.rw-r--r-- david staff  67 KB Tue Dec 13 14:25:00 2022  CV David Kuda.pdf
.rw-r--r-- david staff 1.1 MB Sun Jan  8 22:59:51 2023  Hoare78.pdf
.rw-r--r-- david staff 280 B  Fri Jan 20 18:34:17 2023  main.go
.rwx------ david staff  33 B  Tue Jan 24 22:17:30 2023  script.sh

# you can split the first col into four groups:
| (1) d | (2) rwx | (3) r-x | (4) r-x |

# 1: d represents a directory, . represents a file
# 2: permissions for the user
# 3: permissions for the group
# 4: permissions for all others
```

## WIP: MORE TO COME :)

## Path

## Sourcing scripts

## The Man Pages


STDIN STDOUT. Connect small and specific tools that manipulate input and produce output with shell scripts.

Everything I learned about shell scripts

When Python was released in the 1990s, it was way different to today's version. Python has, obviously, continously improved over time. From the 2000s, the Python team and community have published "Python Enhancement Proposals" (in short: "PEP").

Today I want to present a few of them. By following them, your code will instantly become better: Easier to read and understand (for your future self and others), and easier to maintain.

{{ toc }}

## PEP20: The Zen of Python
Although this is not exactly to follow literally, it expresses what “Pythonic Thinking” is. It’s somewhat amazing that this is referred to as an “Easter Egg”, since in just 20 lines, it conveys so much of what makes Python so accessible.

Open your python, and type `import this` — here is what you will get:

```text
>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
```

## [PEP8](https://peps.python.org/pep-0008): Standard Python Style Guide
Although I nowadays use the [`black` code formatter](https://black.readthedocs.io/en/stable/), I learned a lot of “Pythonic” code style from [PEP8](https://peps.python.org/pep-0008). And I instantly recognize when someone coded without being aware of PEP8.

Some of the guidelines I encountered disregarded are for instance the [order of imports](https://peps.python.org/pep-0008/#imports) or the [line length](https://peps.python.org/pep-0008/#maximum-line-length). 

When you list your imports at the top, PEP8 wants you to start with the builtin packages (e.g. `os` or `json`), then with downloaded packages (e.g. `pandas as pd` or `fastapi`) and thirdly and finally, with your own code that you import (e.g. `from db_client import DBClient`).

```python
# incorrect according to PEP8
import pandas as pd
import os
from db_conn import DBConn
import json

# correct according to PEP8
import json
import os

import pandas as pd

from db_conn import DBConn
```

Second line of order is ordering by the names, alphabetically, e.g. `json` before `os`. 

There is a tool that sorts your imports automatically: `isort`. Just run `isort filename.py` or `isort src/` and it will update all imports in your files. You could run isort before each commit. You could even setup precommit hooks that will check if your sorts are ordered correctly before accepting a commit.

When it comes to line length, [PEP8 advises 79 characters](https://peps.python.org/pep-0008/#maximum-line-length). This is less than most people use, and less than [black default line length of 88](https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#line-length).

By limiting the number of things on a line, you will naturally spread these things over multiple lines.

In the following example, both function definitions are valid, correct python. The first examples has lots of things in one line, the second example spreads them to multiple lines.

```python
# keep many things in one line
# This makes the code hard to read
def encrypt(message: bytes, public_key: RSAPublicKey):
    ciphertext = public_key.encrypt(message, padding.OAEP(mgf=padding.MGF1(algorithm=hashes.SHA256()), algorithm=hashes.SHA256(), label=None))
    return ciphertext

# spread many things to multiple lines
# This is way easier to read and understand
def encrypt(message: bytes, public_key: RSAPublicKey):
    ciphertext = public_key.encrypt(
        message,
        padding.OAEP(
            mgf=padding.MGF1(algorithm=hashes.SHA256()),
            algorithm=hashes.SHA256(),
            label=None
        )
    )
    return ciphertext
```

## PEP484: Type Hints
Python’s dynamic typing system may make reading and understanding Python code obscure and difficult. If a function takes `file` as input, is it a `string`, or a `file_descriptor`? And what does your function return, a dictionary, a list of dictionaries?

Type hints to the rescue, they will remove all ambiguity!

- Close each parameter with a comma, and append the type to it. 
- If it accepts multiple types, separate them with the pipe symbol, e.g. `str | int`
- If it requires a list of strings, define them as `list[str]`
- Also annotate the type your function is going to return.

```python
def answer(q: str) -> int:
    return 42

def rainbow() -> None:
    print("🌈")
```

Generally, type hints have no impact on your code’s performance (but it could if you type hint everything and use certain packages, but at that point, if you need the performance, you would probably pick a strongly typed, compiled language). 

A great benefit is that your code editor will make methods available. Two examples:

If you annotate your arg as type `str`, you will see all available methods in your IDE if you type a dot after it, see here:

![pep484_1](//images.ctfassets.net/pedj0c0bs6fa/4sH2HLacse1c4yF45k0Re3/23f5b3476a964663488fb99e7e02c650/Screenshot_2023-01-16_at_22.19.21.png)

_if you do not give type annotations, your IDE is not going to help you too much._

![pep484_2](//images.ctfassets.net/pedj0c0bs6fa/e37IYsXJmThDzGezN460M/8af8d9694f8c0be128de8a1345b2754e/Screenshot_2023-01-16_at_22.20.05.png)
_if you declare the parameter `s` to be of type `str`, your IDE will start give you tips on what you can do with the parameter. Nice, innit?_

![pep484_3](//images.ctfassets.net/pedj0c0bs6fa/uvJnE4EyGFfUbtFhDZnPc/4250369aa7aca4ae437499b7baeca281/Screenshot_2023-01-16_at_22.24.41.png)
_if  your functions take objects or types / classes, you won’t get any suggestions for their methods._

![pep484_4](//images.ctfassets.net/pedj0c0bs6fa/hMbjLAZ7sS943sFABIPAW/b07676480479a89fb4e1bb44fde0ffb2/Screenshot_2023-01-16_at_22.32.01.png)
_you can just pass the class as type hint. That way, people that use your code will know what to pass as argument, and their IDE will help them to see what methods are available._

## PEP257: Docstrings
After you declare a function, wrap the docstring inside three double quotes on each side.

```python
def answer(question: str) -> int:
    """Give the answer to everything."""
    return 42
```

Summarise what it does in one sentence. Close it with a dot.

If your functions needs more explanation, explain it in multiple lines. Explain what your function does, what argument it takes, what it returns, and what exceptions it could raise. (I learned this from [Google Python style guide](https://google.github.io/styleguide/pyguide.html#381-docstrings)).

Although my example is a bit exaggerated, I think it conveys the point:

```python
def get_json_file_paths(path: str, filename: str = "*.json") -> str:
    """Walk path and yield all file paths that match the filename.

    Args:
        path (str):
            The function will start from this directory and walk 
            all nested directories.

        filename (str, optional):
            Whenever a file matches this string, the function will yield
            the absolute path to this file. Pass for instance "*.json" to
            yield all files that have the .json extension. 

    Yields:
        A string that is the absolute path to the file that was found.

    https://docs.python.org/3/library/os.html#os.walk
    https://docs.python.org/3/library/glob.html#glob.glob
    """
    for dirpath, dirnames, filenames in os.walk(path):
        files_paths = glob.glob(os.path.join(dirpath, filename))
        for file_path in file_paths:
            yield os.path.abspath(file_path)
```

This way, whenever you are using a symbol (function, class), your IDE will show you your docstrings and the type hints. See this example from `os.getenv`:

![getenv_docstrings](//images.ctfassets.net/pedj0c0bs6fa/7366PfHb5GvcN3LE2nIfbl/447acabcdc65b707e4e4c39129dcb51d/getenv_docstrings)

## Conclusion

You have learned how your Python code will become cleaner by embracing the "Zen of Python", by getting familiar with PEP8 or black, and by using type hints and docstrings.

Bonus content if you made this far (thank you!): I can recommend listening to a podcast episode from RealPython where they talked with Lukasz Langa, a core Python team member and the creator of Black: [Real Python Podcast #7: AsyncIO + Music, Origins of Black, and Managing Python Releases](https://realpython.com/podcasts/rpp/7/)

I hope you enjoyed it, happy coding!

By following a few PEPs, your Python code will become become better: Easier to read, easier to maintain. Are you already aware of the Zen of Python, PEP8, Docstrings and Type Hints? If you use them, the people who work with your code are going to love you!

Clean Code Python PEPs

If you write code, chances are high that you use Visual Studio Code as your code editor. And if you do, you might use `code .` to open the current directory.

This models a typical workflow:

```sh
# change into the directory with your code:
cd ~/code/repos/core_services

# open the directory with visual studio code:
code .

# alternatively, open the repo directly without changing into it:
code ~/code/repos/core_services
```

In any case, you always have to either change the directory or type the directory.

There is a shortcut: [`zoxide`](https://github.com/ajeetdsouza/zoxide), a better cd. Zoxide remembers the directories you visit, and enables you to go right into them, without browsing parent directories first.

```sh
# first things first: install zoxide
brew install zoxide

# and init zoxide
echo 'eval "$(zoxide init zsh)"' >> $HOME/.zshrc

# many people alias cd to zoxide, but I don't; instead:
echo 'alias z=zoxide' >> $HOME/.zshrc

# now activate the changes
source $HOME/.zshrc
```

Once you have done that, you can use a shortcut to roam around your filesystem:

```sh
alias z="zoxide"

# zoxide keeps a sqlite3 data base with known directories that were visited,
# so we need to visit a directory at least once:
z ~/code/repos/core_services

# instead of typing out the whole thing,
# you can abbreviate and use the last dir:
z core_services

# pro tip: you do not even have to spell the whole name:
z core # enough to get the right result; at this point, c would be enough, too

# now we can print that directory with "zoxide query <word>":
z query core
# prints the absolute path "/Users/david/code/repos/core_services"

# "zoxide query" will search the database for any directory 
# that matches the given <word>
```

Now we can use code with zoxide in a subshell:

```sh
code $(zoxide query core)
```

Of course, this is longer than the original way, so let's pack the logic into a function.

```sh
# note: cz stands for "code zoxide"
function cz {
  # $1 stands for first argument passed to cz
  path=$(zoxide query $1)
  code $path
}

# once you sourced that function, you can simply type:

cz core

# and this is going to open your repository in visual studio code :)
```

There is a handy shortcut to open your code repositories in visual studio code so that you do not have to type or cd the whole path.

Open Your Code Projects in VSCode From Your Terminal -- Without Typing the Whole Path

[iTerm2](https://iterm2.com/) is a fantastic terminal app for your mac. Many developers use it. In fact, I think most think it's the better alternative to the Mac's standard terminal.

![iterm2-theme-minimal](//images.ctfassets.net/pedj0c0bs6fa/3EhKh14IPEbLFfz1v5cosK/132a27e6f07f06f0b9556712f0cbc759/iterm2-theme-minimal.png)

There are a few settings which will make your iTerm2 experience even better.

### My Most Used Keyboard Shortcuts

I strongly prefer using the keyboard over the mouse or touchpad, and there is a lot that you can achieve in iTerm2 with the keyboard. Here is a list of the keyboard shortcuts that I use the most:

- `cmd + enter` -> activate / deactivate full screen
- `cmd + +` -> zoom in
- `cmd + -` -> zoom out
- `cmd + 0` -> reset zoom
- `cmd + shift + enter` -> maximize current window (especially useful when you use split windows)
- `cmd + t` -> open a new tab
- `cmd + w` -> close focused window
- `cmd + d` -> split window vertically
- `cmd + shift + d` -> split window horizontally
- `cmd + {1..9}` -> Move to tab with number {1..9}
- `cmd + i` -> change name of the tab

### Cycle The Pane Focus With Your Keyboard

A feature that you are maybe already familiar with is splitting the window into panes. You split the pane either with `cmd + d` or `cmd + shift + d`.

![iterm2-split-panes](//images.ctfassets.net/pedj0c0bs6fa/32f2KKRYrgPL0ShnKCjUMb/37b91de6fd69814a723533201bb38ed3/iterm2-split-panes.png)

The default shortcut to cycle the focus of the panes is either `cmd + [` (move focus left) or `cmd + ]` (move focus right). If you use a standard keyboard layout, you would have to move your fingers away from the home row to press these keys.

You may find it way easier (like I do) to remap these actions to `cmd + h` (move left) and `cmd + l` (move right). This is inspired by vim, and for me, intuitive to apply. In vim, you move your cursor to the left with `h` and to the right with `l`. (And for the sake of completeness: To the bottom with `j` and to the top with `k` :) )

open your iTerm2 settings (`cmd + ,`), choose "Keys", click the little plus symbol at the bottom left, select "Previous Pane" or "Next Pane" and assign your favourite key binding so that you don't have to lift your fingers from the home row.

![iterm2-keybindings](//images.ctfassets.net/pedj0c0bs6fa/2w160XEMfYhJceW2frpvbk/64427f0f1afb66f0f6031ba13b52ad53/iterm2-keybindings.png)

Something which is to my individual taste and probably something not everyone will like: I have disabled the dimming of inactive panes (as seen in the picture with the split pane above). If you prefer this, too, you can change it at "Settings/Appearance/Dimming".

![iterm2-pane-dimming](//images.ctfassets.net/pedj0c0bs6fa/6dh3xU9YkptyTsq6hoD3DS/b07a684dae22408c52bfdb51dcca5598/iterm2-pane-dimming.png)

### Change Your Color Preset With `dark()` and `light()`

If you use a light color theme during daytime (e.g. rosé-pine-dawn) and change to a dark color theme in the evening (e.g. rosé-pine), how you maybe might do it now is by opening the preferences menu with `cmd + ,`, clicking on profiles, choose your profile, click on colors, then choose your favorite color theme from the dropdown. Finally, you close the preferences window (e.g. with `cmd + w` or with your mouse on the close symbol at the thop left).

Phew, that are two keyboard shortcuts plus four mouse navigations and clicks! We can do better than that.

I enter the command `dark()` to change my color theme to `rosé-pine`, and the command `light()` to change to `rosé-pine-dawn`.

![iterm2-light-dark](//images.ctfassets.net/pedj0c0bs6fa/22bZUqPsZq6VkSyWbF653i/0259e20dbed55ce4cf8a7830d89aa4e0/iterm2-light-dark.gif)

(As you can see, I couldn't yet figure out how to change the color of the status bar at the bottom, too. For now I accept that the status bar is dark, luckily, it doesn't look too bad.)

To do that, you have to create three custom functions in any of your terminal dot files. If you are using zsh, then you would have to edit the file `~/.zshrc`. The tilde symbol `~` represents your home directory, e.g. `/Users/davidkuda/`. Another synonym for `~` is the global variable `$HOME`, which is normally present in any unix environment. Therefore, the `~/` in `~/.zshrc` means "in your home directory `/Users/davidkuda/`".

The dot in front of the word "zshrc" in `~/.zshrc` means that the file is hidden. If you open your finder and search the file, you will not find it. And if you type `ls ~` (list files in `~`) will also not list a hidden file, unless you add the `-a` option (`ls -a ~`).

So here is the change you need to do:

```sh
change_color_preset() { echo -e "\033]1337;SetColors=preset=$1\a" }
dark() { change_color_preset rose-pine }
light() { change_color_preset rose-pine-dawn }
```

You see the `$1` in the first line. This represents the first argument passed to the function. In the line two and three, you see that you call the function with an argument.

The string that gets executed is an [escape code](https://iterm2.com/documentation-escape-codes.html) that iTerm2 offers to be used as an API to iTerm2.

### Use The Minimal Theme For a Better Look of The Top Bar And Your Tabs

If you use a color theme like the popular [catpuccin](https://github.com/catppuccin/catppuccin) or [rosé pine](https://github.com/rose-pine/rose-pine-theme) (the theme I use) then most likely the Mac standard light / dark theme will interfere with your color scheme. You can avoid that by using iTerm2's minimal theme.

![iterm2-minimal](//images.ctfassets.net/pedj0c0bs6fa/3fqax1jgSULF8N1Q8GqMuf/8543cc021164edc37ac5176327898156/iterm2-minimal.png)

choose "Appearance", from "Theme", select "minimal".

Now let's compare the minimal theme with the regular theme.

#### The Minimal Theme

Notice how well and seamlessly the tabs at the top integrate with your color scheme! wonderful, delightful, joyful. Love it. :)

![iterm2-theme-minimal](//images.ctfassets.net/pedj0c0bs6fa/3EhKh14IPEbLFfz1v5cosK/daaa8131059aef8d73a42b25109d42b0/iterm2-theme-minimal.png)

#### The Regular Theme During Daytime (Light Mode)

Notice how much of a break this leads too! A shame that this is the default and the way more beautiful looking minimal theme is so hidden and has to be discovered. 

![iterm2-theme-regular-light](//images.ctfassets.net/pedj0c0bs6fa/3xI0JLCOvqKdiZls8xpZM9/4b73134a4ceef47a0ac6f54c4846c63b/iterm2-theme-regular-light.png)

#### The Regular Theme During Night (Dark Mode)

I find this even worse than the light mode, since the dark gray of the Mac's dark mode isn't really working so well with the dark theme of the color scheme.

![iterm2-theme-regular](//images.ctfassets.net/pedj0c0bs6fa/1ZEDnm2sNAC85UdY9TARTa/0e802006c16efea537c49f7ff2e877ad/iterm2-theme-regular.png)

### Activate The Status Bar

You can add system information to the bottom of your terminal by activating the "status bar". You may have noticed my status bar in the screenshots above. 

To do so, open your iTerm2 settings with `cmd + ,`, from Profiles, choose yours, select "Session", and at the very bottom, you find the radio button to activate the status bar (wow, it's really well hidden").

![iterm2-statusbar-radio](//images.ctfassets.net/pedj0c0bs6fa/4vdNuRmrhK5ic1mg3q3Fmw/74007552454d356467987be7cce7b263/iterm2-statusbar-radio.png)

### Version Control Your iTerm2 Config

If you ever happen to change to another mac, it would take you long to change all the settings again to your likings (color theme, keymaps, status bar, etc.).

Your configuration is probably located at `~/.config/iterm2/`. The tilde symbol (~) stands for your home directory, e.g. `/Users/davidkuda/` and is synonymous to the environment variable `$HOME`.

Normally, you would encounter folders in this directory, but iTerm2 uses a .plist file and has a rather unusual but specific name, namely: "com.googlecode.iterm2.plist".

![iterm2-config](//images.ctfassets.net/pedj0c0bs6fa/29NS9iG3zAFGwRRYqtmoKz/64fb322a0bf517802deac44cde5a07ca/iterm2-config.png)

```sh
cd ~/.config/
ls -1a # make sure you see the file
git init
git add com.googlecode.iterm2.plist
git commit --message "version control my iTerm2 config"
# I have left the "git add remote" etc out and leave that up to you.
# Once on a new mac, restore by copying file to this folder.
```

### Change The Names of Your Tabs

You can change the name of the current tab by pressing `cmd + i`. Enter a new session name and hit enter, et voilà, your tab is renamed.


iTerm2 is a fantastic terminal app for your mac. There are a few settings which will make your iTerm2 experience even better. For example: Use a minimal design, or activate the status bar.

Settings For a Better iTerm2 Experience

### List Schemas in Database
```sql
select 
  s.nspname as table_schema,
  u.usename as owner
from pg_catalog.pg_namespace s
join pg_catalog.pg_user u on u.usesysid = s.nspowner
order by table_schema;
```
source: https://dataedo.com/kb/query/amazon-redshift/list-schemas-in-database

### List Tables of Schema

```sql
SELECT table_name
FROM information_schema.tables
WHERE table_schema = '{{ SCHEMA }}'
      AND table_type = 'BASE TABLE'
ORDER BY table_name;
```
From https://dataedo.com/kb/query/amazon-redshift/list-of-tables-in-schema

### List Columns of Table

```sql
SELECT column_name
FROM information_schema.columns
WHERE
    table_schema = '{{ SCHEMA }}'
    AND table_name = '{{ TABLE_NAME }}';
```

Often I need to check what tables there are in a schema or what columns there are in a table. In this page, I keep track of these queries.

Common Redshift Queries

Let's say a person from the US is meeting a person from Greece in Zurich, and they only have a weekend to spend there. What activities would someone from Zurich recommend to help the two have a great experience?

### Walk from one side of the river to the other
Start at Bahnhof Enge and walk until you reach Bahnhof Tiefenbrunnen.
![zurich enge tiefenrunnen](//images.ctfassets.net/pedj0c0bs6fa/4a3jDMxDNWl9fSCtvVJsbB/4c2bc4892d893524d8a7f2d0b96ba443/zurich_enge_tiefenrunnen.png)

### "Schanzengraben Promenade": Walk from Zurich Main Station to the Lake Along the Sihl River

Schanzengraben Promenade is a beautiful path to walk by the river "Sihl". The best way to enter the path is from the bridge called the "Gessnerbrücke", right on the other side of the bar "Rio". (The Rio bar is also a great place to have a coffee / beer /drink :) )

![SchanzengrabenPromenadeMap](//images.ctfassets.net/pedj0c0bs6fa/3oZqSW0g7MYgnd8lDGCrNo/a3d90d84ec870565654da9627b0a23b9/SchanzengrabenPromenade.png)

### Walk around Lindenhof

Lindenhof is on top of a hill in the middle of the old town of Zurich. beautiful buildings, nice views. You can start from Zurich Hauptbahnhof.

![maps-lindenhof](//images.ctfassets.net/pedj0c0bs6fa/5cGyXQr2Q5o46aPCuWJvTF/500e6f06168d1075488a21cd03031b38/Screenshot_2022-09-03_at_12.14.35.png)

### Swimming at Letten

Start from Zurich HB Central Station, go to the "Dynamo", stop if you want at "Chuchi am Wasser" for food or a beverage, and continue to walk alongside the river until you reach a place you like for swimming. Locals usually like to swim at the Letten (you can't miss it when you walk there :) ).

![letten](//images.ctfassets.net/pedj0c0bs6fa/6ALPM8XtDYShpmOmBqPaVe/72b60090dc19272433c9fbb4fc162f3b/ZurichDynamoLetten.png)

### Hiking at Üetliberg

You reach the start of the hike in minutes in Zurich. You walk uphill, have good trails, forests, and a fantastic view of Zurich. [Further Information](https://www.zuerich.com/en/visit/attractions/uetliberg)

### Coffees

My favourite places to have coffee  in Zurich are these:

- [Mame](https://mame.coffee/) is the absolute best coffee in Zurich (or perhaps in the world, although hard to compete with all the great coffee  in Berlin :) ) (there are two Mames in Zurich, both are at good locations)
- [Boreal](https://www.borealcoffee.ch/cafes/zurich/?sl=en) 
- [Kraftwerk Café & Food](https://www.kraftwerk.coffee/)

### Drinks

- [Rimini Bar](https://www.zuerich.com/en/visit/restaurants/rimini-bar) (nearby Lindenhof)
- [Rote Fabrik](https://rotefabrik.ch/de/kontakt/#/) (somewhat nearby Bahnhof Enge)

### Wellness at Hürlimann Spa
If you enjoy wellness, you will have a wonderful time at [Hürlimann Spa](https://www.aqua-spa-resorts.ch/de/huerlimannbad-spa-zuerich).


If you have one weekend in Zurich, what are cool activities you can do there?

Activities for a Weekend in Zurich


before:
![terminal-1](//images.ctfassets.net/pedj0c0bs6fa/rAviYbh0QWqzlDslRxlZR/a16f5a53c6e0f515db4fc087efc646d5/1.png)

after:
![terminal-finale](//images.ctfassets.net/pedj0c0bs6fa/4J7ceI5wMX0d0cpGkXgSUl/de7aed8d14df97114dfaeac3772b67b4/finale.png)

This is how your Terminal looks in the beginning:

![terminal-1b](//images.ctfassets.net/pedj0c0bs6fa/nlP5ifpqw8jtAN25I399c/38bddcc0f2c57a2ecfbd699786aeeac1/1b.png)

Step 1: Install [iTerm2](https://iterm2.com/index.html) and [oh-my-zsh](https://ohmyz.sh/#install)
![terminal-2](//images.ctfassets.net/pedj0c0bs6fa/Tvb9mVvXkXTe4Wiob5mBm/dfac036d3b46915cf0d3b6da74795216/2.png)

Step 2: Install a nice iterm2 color theme, e.g. [rosé pine](https://github.com/rose-pine/iterm)

![terminal-3](//images.ctfassets.net/pedj0c0bs6fa/DxUlrNTfs6Su4sAtHB5ek/61679667332a9680187f44a9bcf08947/3.png)

Step 3: Install [Powerlevel10k](https://github.com/romkatv/powerlevel10k)

Step 4: Install further nice terminal apps
- [zsh autosuggestions](https://github.com/zsh-users/zsh-autosuggestions)
- [zsh syntax highlighting](https://github.com/zsh-users/zsh-syntax-highlighting)
- [lsd](https://github.com/Peltoche/lsd) (a colorful ls output)

Step 5: in iTerm2: Turn on the status bar and use the the minimal theme

![terminal-finale](//images.ctfassets.net/pedj0c0bs6fa/4J7ceI5wMX0d0cpGkXgSUl/de7aed8d14df97114dfaeac3772b67b4/finale.png)


Go from naked and boring terminal to a fancy and good looking one on your Mac.

Make your Terminal look awesome

I use my iPhone to take pictures. Although they are all available in photos, I still tend to send those pictures to my macbook with airdrop. This could be, for instance, when I am sending an email and when I want to attach a picture to this email.

When you send your pictures from your iPhone to your Macbook with airdrop, they are in the format “HEIC” (high efficiency image file). I mostly need jpeg.

Another problem is that the file size is too large. Therefore, I usually reduce the resolution of the pictures, and I do it with several clicks with “Preview”. 

### Transforming images with “sips”

Macs have a terminal application called “sips” with which you can transform image files. (See: “[sips MAN page Man Page - macOS - SS64.com](https://ss64.com/osx/sips.html)”)

#### Change file format to jpeg

 If, for example, I have the image `IMG_42.heic` on my desktop, I can transform it to jpeg with sips like so:

```sh
cd ~/Desktop/
sips \
  --setProperty format jpeg \
  --out IMG_42.jpeg \
  IMG_42.HEIC
```

#### Change file size
You can add an additional option to reduce the width of the picture. This will reduce the size of the file.

```sh
sips \
  --setProperty format jpeg \
  --out IMG_42.jpeg \
  --resampleWidth 1080 \
  IMG_42.HEIC
```

#### Change multiple files at once
What if you had multiple pictures that you want to transform? Doing it file by file would be cumbersome. We need a script! Here it is:

```sh
# script.sh
for file in $( ls | grep -i .heic );
do
  name=$( echo $file | cut -f1 -d. )
  sips \
    -s format jpeg \
    --resampleWidth 1080 \
    --out $name.jpeg \
    $file
done
```


Transform .heic pictures on mac with sips in your terminal

Blogging is for ever on my mind. With a full-time job and with being a parent, having the time to do side projects is not always easy. If you see [the commit history of this website,](https://github.com/davidkuda/blog-kuda-ai/commits/main?after=7d55895ce7bac97ea6be48e1ec68b2718c88d280+104&branch=main) you will notice that I started working on this website already in [May 2021](https://github.com/davidkuda/blog-kuda-ai/commit/7f9f10df91a7cdb2f1d9d136665f216216cbd482), almost a year back. That was one month prior to the birth of our daughter. Now is March 2022, and I finally invested more time into it.

### Why I am building the website myself

There are plenty of opportunities to write and publish (e.g. medium, or blogs based on jekyll). If you build something, you usually want it to be used. If you have users, you need great experience. That's at least what I believe. And this website is a good chance to learn about many things:

- Develop web experiences
- Learn web frameworks (React, NextJS)
- Understand how to deploy your frontend (Vercel makes this very easy)
- Understand how the frontend interacts with backend and other services
- Learn how to manage the content and data of a website (Contentful makes that very easy)
- Learn how to build and deploy microservices / backends

You will find the source code of this blog on GitHub: https://github.com/davidkuda/blog-kuda-ai

### The Tech Choices

- JavaScript & React
- Next.js Static Site Generation
- Deployments on Vercel
- Contentful

I like the idea of the "JAM-Stack": JavaScript, APIs and Markup. You have one html page, and you populate the data by calling apis with JavaScript. 

The concepts of combining HTML with JavaScript in JSX was imo an ingenious invention. In the past I've built websites ([example](https://github.com/davidkuda/travel-app/blob/master/src/client/js/markup.js)) where I have written long multiline html strings in JavaScript and used those strings to update the UI. JSX makes this experience very intuitive, and that's for sure one of the things I appreciate. 

So I choose a JAM-Stack framework based on React: [NextJS](https://nextjs.org/). NextJS is a great tech company, for instance, I enjoyed watching the launch of [NextJS 12](https://www.youtube.com/watch?v=dMBYI7pTR4Q). 

NextJS offers many [starter projects](https://nextjs.org/examples) with starter code. It makes some complicated things such as routing easy.

One popular concept that I have noticed was "Headless CMS". If you google it, you will find many results. The demand must be very high, given how many suppliers there are. 

To manage my content, I have chosen [Contentful](https://www.contentful.com/). I think that Contentful is one of Berlin's greatest tech companies. On many websites that I visit, I check out how the website was built. And I have noticed that Contentful has been used on many great websites, e.g. [codecademy](https://www.codecademy.com/learn) (where my coding journey started in 2020).

I first thought that I'd keep my markdown files locally. Calling local files from the browser is normally a pain. Next.js makes it actually very simple to store data in local files. Nevertheless, I thought it would be better to use a backend for the content.

### Deploying The Website

Vercel makes deployments super easy. I signed up to [www.vercel.com](https://vercel.com) with my GitHub account. On vercel, I choose a GitHub repository that I want to deploy. The deployment starts. Everytime I push new commits to that repo, the deployment starts again.

![deployments-on-vercel](//images.ctfassets.net/pedj0c0bs6fa/60WCBmGYAZPuTYDbQsL80U/a9413f759ffe016557514fcb26621399/deployments-on-vercel.png)

There is one thing which is not exactly ideal yet. When I add new content, the new content is not going to be rendered, until I manually trigger a new deployment. That's because of static side generation. The content is fetched at build time and then served as static files. The content is thus not fetched "live" from contentful. That's why redeployments are necessary. 

Luckily, it is very easy to deploy. First, you [install the vercel cli](https://vercel.com/docs/cli#introduction/installing-the-cli) with `npm i -g vercel`. Then, you `cd` into the directory of your project and enter `vercel`. And it's done.

So the deployment still requires some manual effort, but that's acceptable.

### Writing Content

To write content, I log in on www.contentful.com. On Contentful, you create "Content models". With content models, you define the fields of your content. For instance, this blog post has the following fields:

![writing-content-on-contentful](//images.ctfassets.net/pedj0c0bs6fa/TGBL0o9rjJiiMib3gjbDZ/1a2446246daf70ce6a0d82629db2545a/writing-content-on-contentful.png?w=800)

- id: A unique string without spaces that you see in the url
- Creation Date
- Title
- Category (Code, Guitar, or Life)
- Tags (e.g. #python, #react, etc.)
- Description (The preview-text that you will see when you post the link in a chat such as in WhatsApp)
- Body: The text content written in markdown

This post is about the background of the development of this website.

Building kuda.ai


Sometimes some columns carry a list as value. Sometimes a nested dictionary. And sometimes we want to automatically expand and flatten those lists and dictionaries into a larger dataframe. 

You can use [pandas.DataFrame.explode(pandas.DataFrame)](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.explode.html) on lists. This will transform the list into rows. And you can use [pandas.json_normalize(pandas.DataFrame)](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html?highlight=json_normalize#normalization) which will transform nested dictionaries into columns. You will need to drop the column that carried the nested dictionary and [merge](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.merge.html?highlight=merge#pandas.DataFrame.merge) with the flattened dataframe. 

Now you only need to scan if the value of a column is either a list or a dict and apply the corresponding method to unpack the values. The following code example shows how you can automatically detect a collection type and unnest / unpack these columns. 

```python
def unpack_collection_types(df: pd.DataFrame):
  """Explodes all lists and flattens all nested dicts of a df."""

  for col, data in df.iloc[0].items():

      if isinstance(data, list):
          df = df.explode(col, ignore_index=True)
          return unpack_collection_types(df)

      if isinstance(data, dict):
          df_flat = pd.json_normalize(df[col])
          df = df.drop(columns=[col])
          df_combined = pd.merge(
            df, df_flat, left_index=True, right_index=True, suffixes=('', f'({col})'))
          return unpack_collection_types(df_combined)

  return df
```

Flatten nested values in a pandas DataFrame

With the help of two modules from [Python's standard library](https://docs.python.org/3/library/) we will build a functions that can walk a directory and yield the name of any file that matches your criteria. We will use [the os module](https://docs.python.org/3/library/os.html) and [the glob module.](https://docs.python.org/3/library/glob.html) You could for example recursively iterate a directory and yield all files that end in `.json`.

### Walk directories and yield all json files

```python
import os
import glob

def get_json_file_paths(path: str, filename: str = "*.json") -> str:
    """Walk path and yield all file paths that match the filename.

    Args:
        path (str):
            The function will start from this directory and walk 
            all nested directories.

        filename (str, optional):
            Whenever a file matches this string, the function will yield
            the absolute path to this file. Pass for instance "*.json" to
            yield all files that have the .json extension. 

    Yields:
        A string that is the absolute path to the file that was found.

    https://docs.python.org/3/library/os.html#os.walk
    https://docs.python.org/3/library/glob.html#glob.glob
    """
    for dirpath, dirnames, filenames in os.walk(path):
        files_paths = glob.glob(os.path.join(dirpath, filename))
        for file_path in file_paths:
            yield os.path.abspath(file_path)

if __name__ == "__main__":
    paths_to_json_files = get_json_file_paths(
        "~/projects/analyse_economic_data/data",
        "*.json"
    )
```
