yrzam/findb
1
0
Fork 0
mirror of https://github.com/yrzam/findb.git synced 2025-04-11 06:12:29 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

refactor

Browse Source
This commit is contained in:
yrzam 2024-02-26 03:06:07 +04:00
parent 3d59720cc4
commit cda89fd521
3 changed files with 615 additions and 521 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

400
README.md
View File

@ -22,7 +22,7 @@ This repository contains the SQLite schema and scripts to handle financial data.
## Schema
Below you can find the short summary with usage notes for each table. Only columns that need clarification are described. For the complete structure please see [DDL](./schema.sql).
Below you can find the short summary with usage notes for each table. Not all details are described. For the complete structure please see [DDL](./schema.sql).
### fin_assets (table)
@ -30,8 +30,13 @@ Below you can find the short summary with usage notes for each table. Only colum
A **Financial asset** is something you can track a balance of. It should be fungible and tradable.
```
is_base - defines whether this is a main unit of measure of your portfolio. Exactly one row must have this set to true
...
id (pk)
code (no whitespaces, uppercase, unique per type text not null)
name (text)
description (text)
type_id (fk fin_asset_types not null)
is_base (boolean as integer not null) - whether this is a main unit of measurement of your portfolio. Exactly one row must have this set to true
is_active (boolean as integer not null)
```
> Example: US Dollar.
@ -39,7 +44,12 @@ is_base - defines whether this is a main unit of measure of your portfolio. Exac
### fin_asset_types (table)
A **type of financial asset** (also known as an asset class) describes the nature of the financial assets.
A **type of financial asset** (also known as an asset class) describes the nature of the financial asset.
```
id (pk)
name (text unique not null)
```
**The number of records should not exceed 8 due to presentation reasons.**
@ -50,59 +60,63 @@ A **type of financial asset** (also known as an asset class) describes the natur
**Financial storage** represents a place where assets are kept.
> Examples: savings account at a specific bank or broker, bag at home, cryptocurrency wallet.
```
id (pk)
name (text unique not null)
description (text)
is_active (boolean as integer not null)
```
> Examples: savings account at a specific bank or broker, virtual debt account, bag at home, cryptocurrency wallet.
### fin_assets_storages (table)
Join table. Financial storage can hold many assets, and an asset can be held in many storages.
Join table. Financial storage can hold many assets, and an asset can be held in many storages. These intersections must be unique.
```
allocation_group_id - shows which allocation group does a specific asset stored in specific storage belong to
priority - used for sorting balances view and possibly other things
...
id (pk)
asset_id (fk fin_assets not null)
storage_id (fk fin_storages not null)
priority (integer unique) - used for sorting balances view and possibly other things
allocation_group_id (fk fin_allocation_groups) - shows which allocation group does a specific asset stored in specific storage belong to
```
### fin_asset_rates (table)
Stores historical exchange rates of financial assets.
```
rate = [asset_id value] / [base asset from fin_assets value] at [date]
...
id (pk)
datetime (datetime as text not null)
asset_id (fk fin_assets not null)
rate (real not null) - [asset value] / [base asset value]
```
**Rates should be up-to-date on the last day of each month.**
### current_fin_asset_rates (editable view)
Shows current exchange rates. Inactive assets are skipped. If you modify something, a new rate will be saved with a `date` of the current day.
> edit operations: update, insert
```
{asset_type, asset} - lookup tuple
rate - value
other columns - ignored
```
Generally it is desired to know the exchange rate of an asset right before transaction or balance snapshot, although one may also use deep historical data for the analytical purposes.
### fin_transactions (table)
Stores historical transactions. Transaction is an action that leads to a balance change in exactly one place.
Exchanges and self-transfers should be represented by two transactions, both having `is_rebalance` flag in the category. As for now, these two are not linked together due to the homogenous nature of the financial assets.
```
asset_storage_id - points to storage and asset that took part in in transaction, direction is determined by the sign of [amount]
amount - numeric, can be negative
reason_fin_asset_storage_id - (optional) indirectly points to the financial asset, due to which transaction has occurred. This must be not manipulation with the asset itself, but the byproduct of its ownership
reason_phys_asset_ownership_id - (optional) indirectly points to the physical asset, due to which transaction has occurred. This must be not manipulation with the asset itself, but the byproduct of its ownership
...
id (pk)
datetime (datetime as text not null)
description (text)
asset_storage_id (fk fin_assets_storages not null) - points to storage and asset that took part in the transaction
amount (not 0, real not null) - value, direction is determined by the sign
category_id (fk fin_transaction_categories not null)
reason_fin_asset_storage_id (fk fin_asset_storages) - see below
reason_phys_asset_ownership_id (fk phys_asset_ownerships) - see below
```
**Transactions should be up-to-date on the last day of each month, as they are matched with balances. For the initial data import, please create pseudo transactions of a category that has a flag `is_initial_import`. Transactions can be grouped into large blocks that are consistent with the overall balance delta.**
`reason_*` should point to either a financial or a physical asset owned by person if two conditions are met:
1. Transaction occurred because of that ownership
2. Title of that ownership was not directly or indirectly affected by the current transaction.
You may group transactions into batches if it is impossible to log them all.
### fin_transaction_categories (table)
@ -112,77 +126,224 @@ A **transaction category** describes the logical sense of the transaction.
Transaction categories must form a hierarchy with only one root. If a certain flag (starting with `is_`) is set to true on a category, it must be set to true on all of its child categories. The integrity is ensured via triggers that throw exceptions.
```
is_passive - whether the income or expense is passive (see definition below)
is_rebalance - whether the transaction is a part of self-transfer / exchange
is_initial_import - whether the transaction is an upload of the existing assets for accounting (thus it is neither active nor passive income/expense)
parent_id - reference to a category that is a superset of the current one
min_view_depth - in the flattened representation, category shall not appear on a level lower than N, N>=0. Parent category takes multiple levels instead
...
id (pk)
name (text unique not null)
is_passive (boolean as integer not null) - see below
is_initial_import (boolean as integer not null) - whether the transaction is an upload of the existing assets for accounting
parent_id (fk fin_transaction_categories) - reference to a category that is a superset of the current one
min_view_depth (integer not null) - in the flattened representation, category shall not appear on a level lower than N, N>=0. Parent category takes multiple levels instead
```
Income or expense is considered passive if two conditions are met:
Income or expense is considered passive if three conditions are met:
1. It occurred because of some ownership title of which did not change because of the current transaction.
2. * For gains, there are no severe non-monetary losses associated with the transaction reason.
* For losses, there are no severe non-monetary gains associated with the transaction reason.
1. It occurred because of some ownership
2. Title of that ownership was not directly or indirectly affected by this transaction
3. * For gains, there are no significant non-monetary losses associated with the transaction reason.
* For losses, there are no significant non-monetary gains associated with the transaction reason.
> For example, paying tax for the property a person lives in is not a passive loss, although paying it for a property that they lend is a passive loss.
> Examples of transaction categories: expense, salary transfer, rent payment, self transfer or exchange, dividends payout.
Any transaction of a category with `is_passive` must link to a `reason_*` asset, because conditions for setting a `reason_` on transaction are a subset of conditions for `is_passive` of a category. Although a transaction with `reason_*` may be classified as non-passive if it implied some non-monetary gains or losses. This distinction might be useful in evaluation of the overall impact of the asset.
### latest_fin_transactions (editable view)
Shows the latest transactions.
All fields except for pseudo-id are editable. There is also one special column `adjust_balance` - it allows to auto-update `balances` with the amount of the current transaction. It must be set explicitly during any operation (`insert` or `update`) that you want to affect balances. An error will be thrown if transaction's date is not the current date. **Please be cautious: while this option is convenient, used wrongly it may mess up your balance. Verify balances.**
> edit operations: update, insert, delete
```
{pseudo_id} - lookup tuple
amount, date, category, reason_phys_asset - values
{asset_type, asset_code, storage}, {reason_fin_asset_type, reason_fin_asset_code, reason_fin_asset_storage} - value tuples
adjust_balance - pseudo-column, if set to true current operation will be auto-reflected in balances. Works only with transactions of the current day
other columns - ignored
```
> Examples of transaction categories: expense, salary transfer, rent payment, self transfer or exchange, dividends payout.
### balances (table)
Stores historical balances.
**Balances should be up-to-date on the last day of each month, as they are matched with transactions and rates.**
Balance is a verified snapshot of the amount of some asset stored at an asset storage at a given time. Therefore, balance entry may appear anytime without prior transaction activity, and transaction does not create an obligation to update balance right after it happened. Balance is consistent with transactions as long as transaction delta before balance `datetime` equals the balance value: `[balance] = [sum amount of txs where tx datetime < balance datetime]`. Amount can be negative.
```
id (pk)
datetime (datetime as text not null)
amount (real not null)
asset_storage_id (fk fin_asset_storages not null)
```
Whenever possible, balance should be queried directly from this table instead of aggregating transactions.
One may argue that storing balances separately is a bad practice because it causes denormalization and data inconsistency. However, it is a deliberate choice to store conflicting data, as such data is loaded from external sources. Purpose of this project is to offer a viewpoint on multiple versions of data in order to resolve these conflicts.
### current_balances (editable view)
### balance_goals (table)
A **balance goal** is a plan to have a certain amount of financial assets on a balance for the specific purpose in the future, or keep it there constantly.
It is possible to have multiple goals per asset & storage - to complete all of them balance must be equal to or greater than sum of individual goals. Their progress is counted one by one according to the `priority`.
```
id (pk)
name (text not null)
asset_storage_id (fk fin_asset_storages not null)
amount (real not null)
priority (integer unique not null)
deadline (datetime as text) - shows last desired date of completion
result_transaction_id (fk fin_transactions) - if saving resulted in a transaction, that transaction can be linked here. Such a goal will be considered complete
start_datetime (datetime as text not null)
end_datetime (datetime as text) up to that moment goal is relevant, always relevant if set to null
```
### fin_allocation_groups (table)
This table sets a goal for the financial asset distribution.
Each asset & storage (`fin_assets_storages`) from your portfolio can reference a specific allocation group. Calculation should be performed based on `fin_asset_rates`.
```
id (pk)
name (text not null)
target_share (real not null) - a desired fraction of all assets that the group should take, negative value means a negative target balance
start_datetime (datetime as text not null) - since that moment a rule is appled
end_datetime (datetime as text) - up to that moment (excl) a rule is appled. Applied indefinitely if null
priority (integer unique)
```
`target_share` may be any valid number as it shows proportion. Equal target shares indicate that values of the underlying assets should be the same. For a negative target share, there may exist a positive one with the same value that would compensate corresponding debt. Normalization should happen at a later stage.
> Examples: allocation group "CASH" should have a 5% target share in 2025 Q2
### phys_assets (table)
Represents real-world assets, purchases and other non-fungible (non-interchangeable) things. The intended use case is to track large and important assets, especially ones that generate passive gains and losses.
```
id (pk)
name (text unique not null)
description (text)
```
> Examples: house, apartment rented for a year, commercial property, car
### phys_asset_ownerships (table)
Tracks whether physical asset is owned by a person at a particular moment. One asset may be owned at many time periods, or not be owned at all.
```
id (pk)
asset_id (fk phys_asset_id not null)
start_datetime (datetime as text not null) - since that moment an asset is owned
end_datetime - (datetime as text) - up to that moment (excl) an asset is owned. Owned indefinitely if null
```
Ownership periods for the same asset must not intersect.
### swaps (table)
Provides double-entry bookkeeping for the operations where both sides are tracked. Swap is an internal transfer of value that may happen between same or different assets, possibly of different nature. Swap changes the value allocation between financial accounts or physical items.
```
id (pk)
credit_fin_tx_id (fk fin_transactions)
credit_phys_ownership_id (fk phys_asset_ownerships)
debit_fin_tx_id (fk fin_transactions)
debit_phys_ownership_id (fk phys_asset_ownerships)
```
Therefore, possible operations are:
- fin asset -> fin asset (exchange or transfer)
- fin asset -> phys asset (buy)
- phys asset -> fin_asset (sell)
- phys asset -> phys asset (exchange)
- phys asset -> phys asset + fin asset (exchange with change)
> Examples: transfer between bank accounts, currency exchange, buying some item
### current_balances (view)
Shows current financial balances. Inactive assets and storages are skipped.
If you edit `balance`, a supplied one will be saved with a date of the current day. Upon the insertion of a new row, a record in `fin_assets_storages` is created if needed and the balance is upserted.
> edit operations: update, insert
> operations: select, update, insert
```
{asset_type, asset_code, storage} - lookup tuple
balance - value
other columns - ignored
```
### balance_goals (table)
Financial planning begins there. A **balance goal** is a plan to have a certain amount of financial assets on a balance for the specific purpose in the future, or keep it there constantly.
It is possible to have multiple goals per asset & storage - to complete all of them balance must be equal to or greater than sum of individual goals. Their progress is counted one by one according to the `priority`.
```
deadline - shows last desired date of completion
result_transaction_id - if saving resulted in the transaction, that transaction can be linked here. Such a goal will be considered complete
...
lookup tuple:
asset_type (text lkp fin_asset_types.name not null)
asset_code (text lkp fin_assets.code not null)
storage (text lkp fin_asset_storages.name)
value:
balance (real not null)
info:
pseudo_id
asset_name
base_balance - balance converted to base_asset
base_asset
```
Cancelled goals should be deleted.
### latest_fin_transactions (view)
Shows the latest transactions. All fields except for pseudo-id are editable.
For inserts, there is also one special column `adjust_balance` - it allows to auto-update `balances` with the amount of the current transaction. In such case, a new balance entry with datetime one second after transaction will be created or updated. Works only with current datetime. **Please be cautious: while this option is convenient, used wrongly it may mess up your balance. Verify balances.**
> operations: select, update, insert, delete
```
lookup:
pseudo_id (pk)
values:
amount (real not null)
datetime (datetime as text not null)
category (text lkp fin_transaction_categories not null)
reason_phys_asset (text lkp phys_assets via phys_asset_ownerships at datetime)
value tuples:
1.
asset_type (text lkp fin_asset_types.name not null)
asset_code (text lkp fin_assets.code not null)
storage (text lkp fin_asset_storages.name)
2.
reason_asset_type (text lkp fin_asset_types.name not null)
reason_asset_code (text lkp fin_assets.code not null)
reason_storage (text lkp fin_asset_storages.name)
info:
asset_name
special:
adjust_balance (boolean as integer, insert only) - pseudo-column, if set to true current operation will be auto-reflected in balances. Works only if transaction has datetime of the current moment
```
### historical_txs_balances_mismatch (view)
Allows to keep balances consistent with transactions for the analytical purposes.
This view contains a row only if there is a mismatch between transaction delta and balance delta during the last 2 years. It is advised to keep this view empty via adding missing transactions or adjusting balances.
> operations: select
```
info:
start_datetime
end_datetime
storage
amount_unaccounted - difference between balance and transaction delta
tx_delta
balance_delta
```
### current_fin_asset_rates (view)
Shows current exchange rates. Inactive assets are skipped. If you modify something, a new rate will be saved with a `datetime` of the current moment.
> operations: select, update, insert
```
lookup tuple:
asset_type (text lkp fin_asset_types.name not null)
asset (text lkp fin_assets.code not null)
value:
rate (real not null)
info:
pseudo_id
base_asset
```
### current_balance_goals (view)
@ -191,78 +352,45 @@ View that shows statuses of all goals (amount left, whether goal is accomplished
Goals that result in financial transactions are hidden. Goal is considered accomplished if there are enough corresponding assets on the balance. It is reversible, thus needs your attention. Accomplished goals are listed at the bottom of the view.
Goal is considered irrelevant and thus not shown if it either resulted in a transaction or the current moment is outside of the goal's datetime range.
### phys_assets (table)
Represents real-world assets, purchases and other non-fungible (non-interchangeable) things. The intended use case is to track large and important assets, especially ones that generate passive gains and losses.
> Examples: house, apartment rented for a year, commercial property, car, blockchain NFT
### phys_asset_ownerships (table)
Tracks whether physical asset is owned by a person at a particular date. Possibly binds ownership status changes to the financial transactions.
One asset may be owned at many time periods, or not be owned at all. Asset belongs to a person if two conditions are met: `start_date<=[current date]` and `end_date>=[current_date] or end_date is null`.
> operations: select
```
start_date - first day of ownership
end_day - (optional) last day of ownership
buy_fin_tx_id - (optional) transaction id, if redeemed in exchange for financial asset
sell_fin_tx_id - (optional) transaction id, if sold for financial asset
info:
is_accomplished
goal
storage
amount_total
amount_left
deadline
```
### fin_allocation_groups (table)
This table sets a goal for the financial asset distribution.
Each asset & storage (`fin_assets_storages`) from your portfolio can reference a specific allocation group.
```
target share - a desired fraction of all your assets that the group should take. This may be any non-negative number
start_date - since that day a rule is appled
end_date - optional, the last day to be covered by the rule
...
```
> Examples: allocation group "CASH" should have a 5% target share in 2025 Q2
### current_fin_allocation (view)
Shows current asset allocation calculated based on your balance and exchange rates. Both current and target shares are displayed.
### historical_monthly_txs_balances_mismatch (view)
Allows to keep balances consistent with transactions for the analytical purposes.
This view contains a row only if there is a mismatch between transaction delta and balance delta during the last 2 years. It is advised to keep this view empty via adding missing transactions or adjusting balances on a monthly basis.
### historical_monthly_balances (view)
Shows monthly balance with source, calculates deltas - total and grouped by asset type.
Data is calculated over the last 10 years with a period of 1 month for the last day of that month.
> operations: select
```
base_balance - total balance, converted to the base asset
base_balance_delta - balance change since the previous month
base_active_delta - delta (gains - losses) for all transactions that are not passive income/expenses and occurred during this month
base_passive_delta - balance change caused by exchange rate fluctuations, rebalancing, passive income/expenses, non-specified transactions
*_by_type - same data but per asset type, represented as a concatenated string
...
```
info:
group
base_balance
base_asset
current_share - calculated as balance / sum(abs(balance))
target_share
```
Sum of `current_share` percentages without sign is always 100. However, negative balance leads to a negative share. Thus the real sum may vary from `-100` to `100`, where `100` means that all accounted balances are positive, `-100` means they are negative, `0` means that sum of negative balances equals sum of positive balances multiplied by `-1`.
## Schema conventions
### Structure
* Schema shall be described in pure DDL. No initial tuples are allowed.
* Each table has an `id` column as a primary key, stated as `INTEGER AUTOINCREMENT`.
* Boolean is `INTEGER` 1 or 0, date is `NUMERIC`.
* Each table has an `id` column as a primary key, stated as `INTEGER AUTOINCREMENT`, all foreign keys are also `INTEGER`s.
* Use strict types, but do not enable strict mode. Boolean is `INTEGER` 1 or 0, datetime is `TEXT`, numeric is `REAL` (unfortunately).
* Enforce unique in constraints, not indexes.
* Editable views have a `pseudo_id` column with unique non-null values so that client software can identify which row is being edited.
@ -283,7 +411,7 @@ base_passive_delta - balance change caused by exchange rate fluctuations, rebala
* Names of the columns that store boolean must start with `is_`
### Common names
* `code` - string identifier that is required, unique in some way per table and contains no spaces. `code` is a static thing used to identify rows upon data edit.
* `name` - identifier just for the display purposes, that can be edited anytime
* `code` - string identifier that is required, unique in some way per table case-insensitively and contains no spaces. `code` is a static thing used to identify rows externally upon data edit.
* `name` - identifier for the display purposes, that can be edited anytime
* `priority` - unique `INTEGER` value for sorting and other purposes
* `is_active` - used to hide non-needed entries from the current representation, keeping them as historical data

717
schema.sql
View File

File diff suppressed because it is too large Load Diff

19
scripts/dump_schema.sh
View File

@ -2,7 +2,24 @@
DB_PATH=$(realpath "$1")
DUMP_PATH=$(realpath "$2")
SQL="select sql||';' from sqlite_master where sql is not null order by type='table' desc, type='index' desc, type='view' desc, type='trigger' desc, name;"
SQL=$( cat <<SQL
select
m.sql||';
'
from
sqlite_master m
left join sqlite_master ptbl on ptbl.name=m.tbl_name
where
m.sql is not null
order by
m.type='table' desc,
m.type='index' desc,
m.type='trigger' and ptbl.type is not 'view' desc,
m.type='view' desc,
m.type='trigger' desc,
m.name;
SQL
)
read -s -p "Password (if any): " PASS; echo '';
if [ -z "$PASS" ]; then