
## What is Evaluation
Evaluating the performance and capabilities (output) of an LLM based application checking various metrics such as:
- Accuracy
- Coherency
- Toxicity
- Many more

## Why
Directly evaluating one specific hosted LLM such as OpenAI's GPT ada may not be particularly useful outside of selecting a provider but evaluating the output of your tailored prompts and the flow of your LLM based application (agentic or otherwise) will help track how your application performs while leveraging an LLM.

The concept is not specific to LLMs as evaluating the the same metrics can be applicable to any ML based application (and any other complex algorithmic based process) but with the trend towards incorporating LLMs these notes are mainly framed around their use with LLMs.

## In CI
- Track the results *after* merging - need to separate what *may* go in vs. what **is** in the production (main) code
- Can use testing frameworks (e.g. jest) to boostrap the flow while developing specific CI actions

## Types
- Reference Free: Evaluate without examples of the expected output
- Ground Truth: Compare expected/reference output to the result

## Approaches
- LLM as judge evaluator = using an LLM to score the output
- Programmatic = using custom logic to check if the LLM output and/or function calls or correct

## Problems to solve
- Which measure(s) to use
- How to run them
- How to ensure consistency between runs

## Terms
| Term                      | Definitions                                                                                |
| ------------------------- | ------------------------------------------------------------------------------------------ |
| cot                       | chain of thought - reasoning                                                               |
| reference-free evaluation | Evaluate a response without examples/reference data                                        |
| ground truth evaluation   | Evaluate a response with an example expected response/data                                 |
| explicit score            | Prompt for a score between two numbers (e.g 1-5)                                           |
| implicit score            | Prompt expecting yes or no (e.g. "Is the submission harmful, offensive, or inappropriate?) |


## Evaluation Types to be Reviewed
- ROUGE (Recall-Oriented Understudy for Gisting Evaluation)
  - Set of metrics comparing n-gram overlaps
  - https://en.wikipedia.org/wiki/ROUGE_(metric)
  - https://aclanthology.org/W04-1013.pdf
- BLEU
  - bilingual evaluation understudy
  - https://en.wikipedia.org/wiki/BLEU
- F-score
  - [harmonic mean](https://en.wikipedia.org/wiki/Harmonic_mean "Harmonic mean") of the precision and recall
  - https://en.wikipedia.org/wiki/F-score
- Perplexity
  - Measure of the confidence a model has in its next word


### Accuracy/Precision/Recall
- Accuracy = correct predictions / total predictions (% correct)
  - Can be misleading if dataset is imbalanced (i.e. only 5% are matches, getting all wrong = 95% accuracy)
- Precision = how often positive predictions are correct = True Positives/ (True Positives + False Positives)
  - Does not account for false negatives
- Recall = how often true positives are correctly identified = True Positives / ( True Positives + False Negatives)
  - AKA: sensitivity or true positive rate
  - Does not account for false positives

## Topics to Research Further
- Grounding
- Relevance
- Efficiency
- Versatility
- Hallucinations
- Toxicity

## References
- [LangChain - Evaluation](https://python.langchain.com/v0.1/docs/guides/productionization/evaluation/)
- [LangChain - Evaluate an LLM Application](https://docs.smith.langchain.com/how_to_guides/evaluation/evaluate_llm_application)
- [Llama Index - Evaluation](https://docs.llamaindex.ai/en/stable/optimizing/evaluation/evaluation/)



LLM Evaluation


## Overview

With TypeScript, there is risk of `<T>` being considered a tag. Therefor when using an arrow function, you need to either provide an trailing comma, a second generic, or an extends.

## The Issue
```ts
// Possible errors:
// ❌ Cannot find name 'React'.
// ❌ Cannot find name 'T'.
// ❌ JSX element 'T' has no corresponding closing tag.
const test = <T>(value: T) => value;
```

## Solution 1 - Trailing Comma
```ts
// ✅
const test = <T,>(value: T) => value;
```

## Solution 2 - Extends Type
```ts
// ✅
const test = <T extends object>(value: T) => value;
```

## Solution 3 - Second Type
```ts
// ✅
const test = <TValueA, TValueB>(valueA: TValueA, valueB: TValueB) => valueA && valueB;
```

TypeScript: Generic on an Arrow Function


## The Problem

When an array is marked as a const (readonly), its type becomes a union of its values.
```ts
const VALID_VALUES = ['a', 'b'] as const;

const isValid = (value: string): boolean => (
  // ❌ Argument of type 'string' is not assignable to parameter of type '"a" | "b"'
  VALID_VALUES.includes(value);
)
```


## Solution 1 - Widen the type

Explicitly set the type to account for other possible values.

**Caveat**: Widens the type across the board.

```ts
const VALID_VALUES: Readonly<string[]> = ['a', 'b'] as const;

const isValid = (value: string): boolean => (
  // ✅
  VALID_VALUES.includes(value);
)
```

## Solution 2 - Assert the type during the check

Cast the constant during the check.

```ts
const VALID_VALUES = ['a', 'b'] as const;

const isValid = (value: string): boolean => (
  // ✅
  (VALID_VALUES as Readonly<string[]>).includes(value);
)
```

TypeScript: Check if an Array Constant Includes a Value


## Install the plugin
Installing the plugin is straight forward - just follow the [README in the GitHub](https://github.com/tauri-apps/tauri-plugin-sql).

## Migrations
This part I was unable to find documentation for but was able to infer how to add them from the code in plugin code ([`plugin.rs`](https://github.com/tauri-apps/tauri-plugin-sql/blob/dev/src/plugin.rs#L277)).

The migrations can be added by specifying the database URL and a vector of migrations:
```rust
tauri::Builder::default()
    .setup(|app| setup(app))
    .plugin(
        tauri_plugin_sql::Builder::default()
            .add_migrations(
                "sqlite:database_name.db",
                vec![Migration {
                    version: 1,
                    description: "setup tables",
                    sql: include_str!("./migrations/001.setup.sql"),
                    kind: MigrationKind::Up,
                }],
            )
            .build(),
    )
    .run(tauri::generate_context!())
    .expect("error while running tauri application");
```

For the migration SQL, I've put them in a folder `./migrations` and just numbered them sequentially.

I'm still new to rust so this might not be the best approach, but to move the migration definitions out of `main.rs` I moved it to a module `db_migrations.rs`:
```rust
use tauri_plugin_sql::{Migration, MigrationKind};

pub fn get_migrations() -> Vec<Migration> {
    return vec![Migration {
        version: 1,
        description: "setup tables",
        sql: include_str!("./migrations/001.setup.sql"),
        kind: MigrationKind::Up,
    }];
}
```
And then in `main.rs` updated the plugin definition to use this function:
```rust
// Include the added module
mod db_migrations;
...
.plugin(
    tauri_plugin_sql::Builder::default()
        .add_migrations(
            "sqlite:database_name.db",
            // Get the migrations using the functin defined in the module
            db_migrations::get_migrations()
        )
        .build(),
)
...
```

## Running Migrations on Launch
This is another part that I was unable to find in the documentation but was able to figure out reading the relevant code in [`plugin.rs`](https://github.com/tauri-apps/tauri-plugin-sql/blob/dev/src/plugin.rs#L296). For each database that you would like to run the migrations for at startup, you can add them to the `preload` option of the `sql` plugin configuration in `src-tauri/tauri.conf.json`:
```json
{
  "build": {
    ...
  },
  ...
  "plugins": {
    "sql": {
      "preload": [
        "sqlite:database_name.db"
      ]
    }
  }
}
```

## References
- [Plugin tauri-plugin-sql GitHub](https://github.com/tauri-apps/tauri-plugin-sql)
- [Tauri plugins GitHub](https://github.com/tauri-apps/plugins-workspace)
- [Tauri configuration docs](https://tauri.app/v1/references/configuration-files)

Setting up the Tauri SQL Plugin


These are the settings that I use in tandem with [ESLint](https://eslint.org/) and [Prettier](https://prettier.io/).

1. Go to `File > Preferences > Settings`
2. Search for `On Save`
3. Under `Editor: Code Actions On Save` click `Edit in settings.json`
4. Set the following:
    ```json
    "editor.codeActionsOnSave": [
        "source.organizeImports",
        "source.fixAll.eslint"
    ]
    ```

This will organize imports alphabetically (`source.organizeImports`) and use `eslint` to fix any auto-fixable errors. Using prettier this will take care of formatting as well.

For organizing imports, I initially was only using `source.organizeImports` but this was removing the trailing commas on imports which conflicted with my prettier settings. By following it with `source.fixAll.eslint`, the prettier rules fix this at the same time.

## References
- [[StackOverflow] Enable trailing comma in vscode](https://stackoverflow.com/questions/66508443/enable-trailing-comma-in-vscode)

VSCode: Auto Format on Save (Prettier)


## Parent Expanding to the Size of Child Elements
```scss
.container {
  display: grid;
  // Fits columns to the available space, with a min width of 100px
  // and a max width of 1 fraction of the container
  grid-template-columns: repeat(auto-fit, minmax(100px, 1fr));
  gap: 8px;
  min-width: 150px;
}
```

## Child Elements Filling Parent
```scss
.container{
  display: flex;
  gap: 8px;
  min-width: 150px;

  .child {
    // flex: {flex-grow} {flex-shrink} {flex-basis}
    // Using flex-basis of 0, all elements have the same starting point
    flex: 1 1 0;
  }
}
```

## References
- [[StackOverflow] How to make flexbox items the same size?](https://stackoverflow.com/questions/29503227/how-to-make-flexbox-items-the-same-size)
- [[MDN] CSS Grid Layout](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Grid_Layout)
- [[CSS-Tricks] Auto-Sizing Columns in CSS Grid: `auto-fill` vs `auto-fit`](https://css-tricks.com/auto-sizing-columns-css-grid-auto-fill-vs-auto-fit/)

CSS Equal Width Elements

- [Window Size Changes](#window-size-changes)
- [Watching a Specific Element for Size Changes](#watching-a-specific-element-for-size-changes)
- [React Hook](#react-hook)
- [References](#references)

----

## Window Size Changes
If it your logic based off of the window size and not specific element sizing, an event listener can be added to the window for the `resize` event with:
```typescript
const callback = (evt: UIEvent) => console.log(evt);
window.addEventListener('resize', callback);
```

When done, be sure to remove the event listener with:
```typescript
window.removeEventListener('resize', callback);
```

## Watching a Specific Element for Size Changes
For cases where a specific element's size needs to be known, a `ResizeObserver` can be used. This is helpful for when the window doesn't resize but a element's size does (e.g. reflow due to content being added or removed).

To watch an element, the following can be used:
```typescript
const testDiv = document.getElementById('test');
const callback = (entries: ResizeObserverEntry[], observer: ResizeObserver) => console.log(entries);
const resizeObserver = new ResizeObserver(callback);
resizeObserver.observe(testDiv);
```

When done watching an element, it can be unobserved using:

```typescript
// Only stop watching a single element
resizeObserver.unobserve(testDiv);

// Or Alternatively, all elements can be unobserved using:
resizeObserver.disconnect();
```

## React Hook
This hook is a general purpose resize hook. It can be used to track the resizing of a specific element or when the window itself is resized.

If `elementToWatch` isn't passed, the whole window will be watched for resize events.

```typescript
import { useEffect } from 'react';

/**
 * Hookup a resize listener to the window or a target element
 * @param {() => void} callback The callback to call on resize (should be memoized with `useCallback` to avoid re-registering each render)
 * @param {boolean} callOnMount Whether to trigger the callback on mount
 * @param {HTMLElement | undefined} elementToWatch The element to watch for size changes
 */
export default function useOnResize(
  callback: () => void,
  callOnMount: boolean = false,
  elementToWatch?: HTMLElement,
) {
  useEffect(() => {
    // Watch a particular element if specified
    if (elementToWatch) {
      const resizeObserver = new ResizeObserver(callback);
      resizeObserver.observe(elementToWatch);
      if (callOnMount) {
        callback();
      }

      return () => {
        resizeObserver.disconnect();
      };
    }

    // If no element is specified, watch for window size changes
    window.addEventListener('resize', callback);
    if (callOnMount) {
      callback();
    }
    return () => {
      window.removeEventListener('resize', callback);
    };
  }, [callback, callOnMount, elementToWatch]);
}

```


## References
- [[MDN] ResizeObserver
](https://developer.mozilla.org/en-US/docs/Web/API/ResizeObserver)
- [[MDN] Window: resize event
](https://developer.mozilla.org/en-US/docs/Web/API/Window/resize_event)



Monitor for Resize in JavaScript

- [Configuration](#configuration)
  - [Redirect HTTP to HTTPS via 301](#redirect-http-to-https-via-301)
  - [Including SSL Certificate for HTTPS](#including-ssl-certificate-for-https)
  - [WebSocket Connections](#websocket-connections)
  - [Load Balancing](#load-balancing)
  - [Basic Full Example](#basic-full-example)
  - [Adding Custom Stream Configs](#adding-custom-stream-configs)
- [Commands](#commands)
- [PID Issue on Some Alpine Linux Docker Images](#pid-issue-on-some-alpine-linux-docker-images)
- [References](#references)

----

## Configuration

### Redirect HTTP to HTTPS via 301
```nginx
server {
  listen *:80;

  server_name k3s.functionalbytes.com;
  server_tokens off; ## Don't show the nginx version number

  location / {
    return 301 https://k3s.functionalbytes.com:443$request_uri;
  }
}
```

### Including SSL Certificate for HTTPS
```nginx
ssl_certificate /secret/keys/proxy.functionalbytes.com.fullchain;
ssl_certificate_key /secret/keys/proxy.functionalbytes.com.key;
```

### WebSocket Connections
See: [[nginx Documentation] WebSocket proxying](http://nginx.org/en/docs/http/websocket.html)
```nginx
location /web_socket_path/ {
    proxy_pass http://backend;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
}
```

### Load Balancing
See: [[nginx Documentation] HTTP Load Balancing](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/)
```nginx
upstream k3s_servers {
  # Round Robin by default - can add "weight=#" after each server for weighting, e.g.:
  # server k3s-s1.functionalbytes.com:80 weight=3;
  # With this set and nothing on k3s-s2 (default = 1), 3 out of 4 requests will be sent to k3s-s1

  # Alternatively one of the following can be uncommented for other load balancing options

  # Uncomment the line below to use the one with the fewest connections
  # least_conn;

  # Uncomment to send requests to the server based on a hash of the first 3 octets of their IP (guaranteeing requests go to the same server unless it is unavailable)
  # ip_hash;
  server k3s-s1.functionalbytes.com:80;
  server k3s-s2.functionalbytes.com:80;
}
```

### Basic Full Example
```nginx
upstream k3s_servers {
  server k3s-s1.functionalbytes.com:80;
  server k3s-s2.functionalbytes.com:80;
}

server {
  listen *:80;

  server_name k3s.functionalbytes.com;
  server_tokens off; ## Don't show the nginx version number, a security best practice

  location / {
    return 301 https://k3s.functionalbytes.com:443$request_uri;
  }
}

server {
  listen 443 ssl http2;

  server_name k3s.functionalbytes.com;
  server_tokens off; ## Don't show the nginx version number, a security best practice

  ssl_certificate /secret/keys/proxy.functionalbytes.com.fullchain;
  ssl_certificate_key /secret/keys/proxy.functionalbytes.com.key;

  # Allow large attachments
  client_max_body_size 128M;

  location / {
    proxy_pass http://k3s_servers;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }
}
```

### Adding Custom Stream Configs
1. Add an include directive to `/config/nginx/nginx.conf`
    > File: `/config/nginx/nginx.conf`
    ```nginx
    stream {
        include /config/nginx/stream-confs/*;
    }
    ```
2. Add the stream configs in `/config/nginx/stream-confs/`, e.g.:
    > File: `/config/nginx/stream-confs/k3s.conf`
    ```nginx
    upstream k3s_servers {
      server k3s-s1.functionalbytes.com:6443;
      server k3s-s2.functionalbytes.com:6443;
    }

    server {
      listen *:6443;

      ssl_certificate /secret/keys/proxy.functionalbytes.com.fullchain;
      ssl_certificate_key /secret/keys/proxy.functionalbytes.com.key;

      proxy_pass k3s_servers;
    }
    ```

## Commands
Reload the nginx config without any downtime (sends `SIGHUP` signal):
```bash
nginx -s reload
```

Verify configuration is valid:
```bash
nginx -t
```

## PID Issue on Some Alpine Linux Docker Images
**Note:** I am currently using `nginx:1.21-alpine` (official) which does not have this issue, but do still see the issue with `linuxserver/nginx:1.20.2` ([linuxserver.io](https://www.linuxserver.io/) image that I was using this on [Unraid](https://unraid.net/))

There is an issue where the PID file isn't created in the alpine linux version of nginx docker image. Running the reload command will result in the following error:
```bash
> nginx -s reload
nginx: [error] open() "/run/nginx/nginx.pid" failed (2: No such file or directory)
```

Two options for fixing this:

1. Create the required directory
    > File: `Dockerfile`
    ```Dockerfile
    ...
    RUN mkdir -p /run/nginx
    ...
    ```
1. Override the PID location
    ```sh
    nginx -g 'pid /tmp/nginx.pid; daemon off;'
    ```

## References
- [[Ubuntu Manpage] nginx](https://manpages.ubuntu.com/manpages/jammy/man8/nginx.8.html)
- [[nginx Documentation] WebSocket proxying](http://nginx.org/en/docs/http/websocket.html)
- [[nginx Documentation] HTTP Load Balancing](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/)
- [[GitHub Issue #185] latest image run nginx problems](https://github.com/gliderlabs/docker-alpine/issues/185)



Configuring Nginx


- [Rough Notes](#rough-notes)
  - [Commands](#commands)
  - [Init Systems](#init-systems)
  - [Service Managers](#service-managers)
- [Important Reading](#important-reading)
- [References](#references)

----

## Rough Notes

- [systemd](https://en.wikipedia.org/wiki/Systemd) is for both system init and service management
- [supervisord](http://supervisord.org/) is only for service managment

### Commands
- The [`service`](https://manpages.ubuntu.com/manpages/jammy/en/man8/service.8.html) command is a "wrapper"/"high-level" command that will call the respective "low-level" manager such as [`systemctl`](https://manpages.ubuntu.com/manpages/jammy/man1/systemctl.1.html)
- The [`service`](https://manpages.ubuntu.com/manpages/jammy/en/man8/service.8.html) command allows for basic interatctions such as `status`, `start`, `reload`, etc.
- The [`systemctl`](https://manpages.ubuntu.com/manpages/jammy/man1/systemctl.1.html) command allows for advanced commands such as checking if a service is enabled (activated when booting): `systemctl is-enabled sshd.service`

**Quick Reference**
- Enable service to run at startup
    ```bash
    sudo systemctl enable sshd.service
    ```
- Disable service from running at startup
    ```bash
    sudo systemctl disable sshd.service
    ```

### Init Systems
- Init systems are those that run in process #1 - responsible for bootstrapping the system
- May also be responsible for starting and managing services (long running, background process)
- Ubuntu used to use [upstart](https://upstart.ubuntu.com/) but switched to [systemd](https://en.wikipedia.org/wiki/Systemd) in version 15.04 (2015)
- Alpine linux uses [OpenRC](https://wiki.alpinelinux.org/wiki/OpenRC)
- `systemd` is linux specific - (BSD distros such as FreeBSD have their own ["BSD-style init"](https://docs.freebsd.org/en/articles/linux-users/#startup))
- Although highly adopted, there is a lot of contreversy around `systemd` (see: [[Reddit] ELI5: The SystemD vs. init/upstart controversy](https://www.reddit.com/r/linux/comments/132gle/comment/c709592/?utm_source=share&utm_medium=web2x&context=3))

### Service Managers
Outside of init systems, there are also dedicated service managers such as [Supervisor (supervisord)](http://supervisord.org/)
> It shares some of the same goals of programs like launchd, daemontools, and runit. Unlike some of these programs, it is not meant to be run as a substitute for init as “process id 1”. Instead it is meant to be used to control processes related to a project or a customer, and is meant to start like any other program at boot time.

*[[supervisord.org] Supervisor: A Process Control System](http://supervisord.org/#supervisor-a-process-control-system)*

- Advantages of using something like `supervisord` are that it is OS agnostic and since it is outside of process #1, processes can be managed by users other than root/sudo

## Important Reading
- [[StackExchange] What is the difference between "init" and "service manager"?](https://unix.stackexchange.com/a/489949)
- [[gentoo wiki] Comparison of init systems](https://wiki.gentoo.org/wiki/Comparison_of_init_systems)

## References
- [[Stack Overflow] Difference between Systemctl and service command](https://stackoverflow.com/a/43539946)
- [[Ask Ubuntu] Difference between systemctl and service commands](https://askubuntu.com/questions/903354/difference-between-systemctl-and-service-commands)

Linux Service Managers and Init Systems


> "In quantum mechanics, wave function collapse occurs when a wave function — initially in a superposition of several eigenstates — reduces to a single eigenstate due to interaction with the external world."

*[Wikipedia: Wave function collapse](https://en.wikipedia.org/wiki/Wave_function_collapse)*

## How WFC Applies to World Generation
- For world generation, each tile has a set of possible types of tiles
  - Until one is chosen, it is in its "superposition" state
- The tile map is procedurally iterated over, randomly selecting one of the possible types for each tile
  - This is the collapse phase
- There are adjacency rules for which tiles can be next to each other, so as tiles are chosen, the set of possible types for adjacent tiles will be reduced (probability space reduced)
  - These rules can be different depending on direction from the tile

## Determining Valid Neighbors
Multiple approaches:
- Map of valid neighbors for each tile / direction
- "Socketing" system where each tile / direction has a socket ID that is used to match valid neighbors
- [Relevant section of "Superpositions, Sudoku, the Wave Function Collapse algorithm" Video](https://youtu.be/2SuvO4Gi7uY?t=265)

## Ideas for Pathing
  - There can be an additional step after each tile is collapsed to check that pathing is still possible and the previous tile can be regenerated on failure until pathing is solvable
  - The adjacency rules could include pathing when reducing neighboring sets (i.e. if there are are walls on all other sides, walls are excluded from the possible set for the remaining tile)
  - The entire world/map could be generated and then checked that pathing is solvable using something like [A*](https://en.wikipedia.org/wiki/A*_search_algorithm) - either breaking down paths or re-generating until solvable

## References

- [Wikipedia: Wave function collapse](https://en.wikipedia.org/wiki/Wave_function_collapse)
- [Video: Why I'm Using Wave Function Collapse for Procedural Terrain | Unity Devlog
](https://www.youtube.com/watch?v=20KHNA9jTsE)
  - [Jump to Summary section](https://youtu.be/20KHNA9jTsE?t=388)
- [Video: Superpositions, Sudoku, the Wave Function Collapse algorithm.
](https://www.youtube.com/watch?v=2SuvO4Gi7uY)
