# emptymalei/powerbi-cli (version 1) Pack digest: `sha256:8dae4a6ef26d595fbf4c4eb4089d67d2ef1b71e186cb78a8c20eb430bec8b2dc`. This file holds the brief, assumptions, obligations and manifest. Each component's contract, schemas and tests are in their own file; read the ones you build. Raw files are under [files/](https://files.symbola.sh/packs/emptymalei/powerbi-cli/files/), and the tarball holds everything. ## Components - [parts/auth-profiles-v1.md](https://files.symbola.sh/packs/emptymalei/powerbi-cli/parts/auth-profiles-v1.md) (27 KB) - [parts/cache-keys-v1.md](https://files.symbola.sh/packs/emptymalei/powerbi-cli/parts/cache-keys-v1.md) (20 KB) - [parts/cached-reads-v1.md](https://files.symbola.sh/packs/emptymalei/powerbi-cli/parts/cached-reads-v1.md) (29 KB) - [parts/config-cache-v1.md](https://files.symbola.sh/packs/emptymalei/powerbi-cli/parts/config-cache-v1.md) (25 KB) - [parts/file-converters-v1.md](https://files.symbola.sh/packs/emptymalei/powerbi-cli/parts/file-converters-v1.md) (16 KB) - [parts/flatten-v1.md](https://files.symbola.sh/packs/emptymalei/powerbi-cli/parts/flatten-v1.md) (15 KB) - [parts/scan-batch-v1.md](https://files.symbola.sh/packs/emptymalei/powerbi-cli/parts/scan-batch-v1.md) (21 KB) - [parts/workspace-entries-v1.md](https://files.symbola.sh/packs/emptymalei/powerbi-cli/parts/workspace-entries-v1.md) (12 KB) ## AGENTS.md # AGENTS.md This is a Symbolon spec pack. Build from it; do not look for the original source. 1. Read `BRIEF.md`: what the product is, its non-goals and priorities. 2. Read `assumptions/`: what you may take as true about the world. Don't build for cases they rule out. 3. Read `obligations.toml`: the decisions you must keep. Obligations marked `open` are yours to choose; ones marked `unchecked` are required although no check covers them yet. 4. Each component's full specification is `components//versions//`: `symbolon.toml` (the description), `schema.py`, `examples.jsonl` and `properties.py`. Where the description and an example seem to disagree, the example is what the original does. **Harness.** An implementation is a directory with `symbolon-impl.toml` (`symbola impl-init --contract components//versions/`). In Python, define `run(input)` returning the contract's output; in any other language, a program that reads one JSON input per line on stdin and writes `{"output": ...}` per line. A component whose `symbolon.toml` says `driver = "cli"` is a command line: build the real program and set `command` to it; symbola runs it once per check. **The product.** If `PRODUCT.md` exists, the pack is a whole product: build it as `PRODUCT.md` describes (its shape, surfaces and flows), with your component builds as its core. Don't build a demo page around the components instead. **Done** means `symbola attest components//versions/ ` reports PASSED for every component and, for a product pack, every flow in `PRODUCT.md` works. Don't special-case the examples: the properties generate other inputs, and the publisher keeps a holdout. Write questions about unclear parts, and the reading you chose, in `NOTES.md` next to your build. **If the spec contradicts itself**, don't code around it. Run `symbola flag #example: "" --dir `, build the reading you think is right, and carry on. ## BRIEF.md # powerbi-cli (`pbi`) `pbi` is a command line tool for people who administer or document a Power BI tenant. They give it a bearer token, and it asks the Power BI REST API (`https://api.powerbi.com/v1.0/myorg`) for workspaces, apps, reports, report users, user access and workspace scans, and saves the answers as JSON files and Excel workbooks, one folder per day if they like. Results can be cached in a folder (local or S3) so later runs can work offline, and saved files can be fed back into later commands (convert to Excel, fetch the users of each report). Its users script it (PowerShell or bash loops, scheduled backups), so what they rely on is the command line, the exit codes, the names and shapes of the files it writes, the profile/config/cache files in `~/.pbi_cli`, and the messages that say why a run failed. Almost every command needs the live API and cannot be checked here; the pack pins what can be checked offline (profiles and settings, the cache, the saved-file converters, the scan config) and records the API calls as decisions. ## Non-goals - Getting a token. The user brings a bearer token; there is no sign-in flow, no refresh, no service principal login. - Changing anything in Power BI. Every call is a read (the one POST asks for a scan). - A graphical interface, a server, or a library API for others to import. - Other clouds (national or sovereign Power BI endpoints) and API versions other than `v1.0/myorg`. - Concurrency control: two `pbi` processes sharing one home directory are not supported. ## Priorities When goals conflict, earlier ones win: 1. Scripts keep working: command names, options, exit codes, file names and the shapes of saved JSON and Excel files do not change. 2. Tokens stay out of sight: kept in the OS keyring when there is one, otherwise in a file only the user can read; never printed or logged. 3. A run fails loudly and says why; one failing workspace or app does not stop a batch that can go on. 4. Work offline from the cache and from saved files whenever the data is already there. 5. Few dependencies, plain files, no hidden state beyond `~/.pbi_cli` and the cache folder. ## Who resolves gaps emptymalei, as the namespace holder. If the pack doesn't settle something, record the question and the reading you chose in `NOTES.md` next to your build; obligations marked `open` in `obligations.toml` are yours to choose. ## PRODUCT.md # powerbi-cli: the product `pbi` is one command line program that reads Power BI's REST API and saves the answers as files. The components in this pack are its core (the parts that can be checked without the API); this page says what the whole program is. ## Shape - A command line program, `pbi`, written in Python (3.10 or later) and installed as a package named `pbi_cli` (`uv sync`, then `uv run pbi`, or `pip install`). It runs on Windows, macOS and Linux; its documentation shows PowerShell and bash loops around it. No server, no daemon, no GUI. - Command line parsing with `click`, nested command groups with `--help` on each, short options like `-t`, `-tf`, `-ft`, `-wn`, `-wi` as listed below. Run with no command it greets and points to `--help`; `pbi version` prints the installed package version. - Tables and Excel files with `pandas` and `openpyxl`; configuration and the scan config with YAML; HTTP with `requests`; file names with `python-slugify`; cache locations through `cloudpathlib` (local folders and `s3://`); the OS keyring through `keyring` when one exists. Logging goes to standard error at INFO level; results and messages go to standard output. - It needs a Power BI bearer token supplied by the user (see Interfaces). Commands marked admin need a token with admin rights. - State lives in `~/.pbi_cli/` (`config.yaml`, `credentials.json`, the legacy `auth.json` and `profiles.json`) and in the optional cache folder. Nothing else is kept between runs. ## Surfaces Setup and local state (offline, checked): - `pbi auth -t TOKEN [-p PROFILE] [-g user|admin]`: store a token as a profile. (`auth-profiles`) - `pbi profile list | switch [NAME] [-g G] | delete NAME [-g G] [--yes]`: manage profiles. (`auth-profiles`) - `pbi config set-output-folder P | get-output-folder | set-cache-folder P | get-cache-folder | enable-cache | disable-cache | set-verify-certificates true|false|PATH | show`: settings. (`config-cache`) - `pbi cache list [-k KEY] | clear [-k KEY] [-v VERSION] [--yes]`: look at and empty the cache folder. (`config-cache`, `cache-keys`) - `pbi version`, and `pbi` alone (a greeting). Reading the API (needs a token and the network; each can also read the cache, see Flows): - `pbi workspaces list [--top N] [-e KIND]... [-f ODATA] [-ft json|excel]... [-tf FOLDER] [-n NAME] [--use-cache|--cache-only]`: admin; all workspaces, with users, reports, dashboards, datasets, dataflows, workbooks expanded by default; prints a table, or saves `NAME.json` and `NAME.xlsx` (default name `workspaces`). - `pbi apps list [-r user|admin] [-tf FOLDER] [-ft ...] [-n NAME] [--use-cache|--cache-only]`: the apps the user can see, or all apps of the tenant (admin); default file name `apps`. - `pbi apps app -a APP_ID [-t FILE] [-ft json|excel]`: one app with its reports and dashboards; prints JSON, or saves it (an Excel file has the sheets of `flatten` kind app). - `pbi users user-access -u USER_ID [-tf FOLDER] [-ft ...] [-n NAME] [--use-cache|--cache-only]`: admin; the artifacts a user can access, as `{"artifacts": [...]}`. - `pbi reports list -g GROUP_ID [-t FILE]`, `pbi reports pages -g GROUP_ID [-r REPORT_ID] [-t FILE]`, `pbi reports export -g GROUP_ID -r REPORT_ID [-t FILE]` and the older `pbi export -g GROUP_ID -r REPORT_ID [-t FILE]`: reports of a workspace, their pages (all reports of the group when `-r` is left out; a report whose pages cannot be fetched is skipped and logged), and the exported report file (without `-t` it prints the size instead of the content). - `pbi workspaces scan initiate IDS... | status SCAN_ID | result SCAN_ID [-t FILE] | get IDS... [-t FILE | -tf FOLDER]`: admin; the workspace-info scan, step by step or all at once, with `--lineage --datasource-details --dataset-schema --dataset-expressions --get-artifact-users`, `--interval` (default 5) and `--timeout` (default 300) for `get`. `-t` and `-tf` exclude each other and `-tf` takes exactly one workspace id (it saves `.json`). - `pbi workspaces scan batch -c CONFIG.yaml`: admin; scans many workspaces one by one from a config file. (`scan-batch`, `workspace-entries`) Working on saved files: - `pbi workspaces format-convert -s WORKSPACES.json -t WORKSPACES.xlsx`: JSON to Excel. (`file-converters`, `flatten`) - `pbi apps augment -s APPS.json -t OUT [-ft json|excel]`: fetch every listed app in full. (`file-converters`, `flatten`) - `pbi reports users -s APPS.json -t OUT [-ft json|excel]`: the same, plus the users of each report. (`file-converters`, `flatten`) - `pbi workspaces report-users -s WORKSPACES.xlsx [-wn NAME]... [-wi SECONDS] [-ft json|excel]... [-t FOLDER] [-n NAME]`: admin; reads the `reports` sheet of the Excel file `workspaces list` saved, keeps rows whose `type` is `Workspace` (and, with `-wn`, whose workspace `name` is listed), fetches the users of each report (`reports_id` column), waiting `-wi` seconds (default 3) between requests, and saves `workspaces_reports_users.json`/`.xlsx` (name with `-n`) with one list of enriched report records per workspace name, flattened by `flatten` kind report_users; prints a summary table without `-t`. Files users touch: `~/.pbi_cli/config.yaml`, `~/.pbi_cli/credentials.json`, the cache folder (`//.json`), saved `*.json` and `*.xlsx` results, scan config YAML files (an example is `examples/scan_config.example.yaml`). ## Flows 1. **First setup.** `pbi auth -t TOKEN -p work -g admin` (and `-g user` for a user token), `pbi profile list` shows ✓ marks and the active profile of each group, `pbi profile switch work -g admin` changes it. Commands then use the right group's active profile. (`auth-profiles`) 2. **Back up all workspaces.** `pbi config set-output-folder ~/PowerBI/backups`, then `pbi workspaces list -ft json -ft excel -tf 2024-01-01` fetches (admin API), caches the result if a cache folder is set (`cache-keys`), and saves `workspaces.json` and a workbook with the sheets users, reports, dashboards, datasets, dataflows, workbooks and workspace (`flatten`) in `~/PowerBI/backups/2024-01-01/`. A JSON saved earlier is converted later with `pbi workspaces format-convert`. (`config-cache`, `file-converters`) 3. **Work offline from the cache.** `pbi config set-cache-folder ~/PowerBI/cache` once; every list command saves what it fetched as a new version. `pbi workspaces list --cache-only` prints the latest saved result without a token or network, `--use-cache` falls back to the API when nothing is cached; `pbi cache list [-k workspaces]` shows keys and versions, `pbi cache clear -k workspaces -v VERSION` removes some. (`cached-reads`, `config-cache`, `cache-keys`) 4. **Who can open which report.** `pbi workspaces list -ft excel -tf ...`, then `pbi workspaces report-users -s workspaces.xlsx -wn "Finance" -wi 5 -tf ...` saves the report users of one workspace; or `pbi apps list -tf ...`, `pbi reports users -s apps.json -t out.json` for apps. (`file-converters`, `flatten`) 5. **Scan many workspaces.** Copy `scan_config.example.yaml`, list workspace ids (with optional names for readable file names), set `target_folder`, run `pbi workspaces scan batch -c scan.yaml`: each workspace is scanned on its own (start the scan, poll its status every `interval` seconds up to `timeout`, fetch the result) and saved as `-.json`; failed workspaces are listed at the end and the exit code is 1. (`scan-batch`, `workspace-entries`) 6. **Look at one thing.** `pbi apps app -a ID`, `pbi reports list -g GROUP`, `pbi reports pages -g GROUP -r REPORT`, `pbi users user-access -u USER`, `pbi reports export -g GROUP -r REPORT -t report.pbix`, `pbi workspaces scan get IDS... --lineage -t result.json`. ## Interfaces - **Power BI REST API**, `https://api.powerbi.com/v1.0/myorg` (hard-coded), with the header `Authorization: Bearer ` on every call. TLS certificates are verified unless the `verify_certificates` setting says otherwise (`app.verify-certificates`). Failed connections and HTTP 500, 502 and 504 are retried up to 5 times with a 0.3 backoff factor. Which endpoint and query parameters each command uses is in the `app.*` obligations. - **Credentials**: bearer tokens per profile name, in the OS keyring (service `pbi-cli`) when it works, otherwise in `~/.pbi_cli/credentials.json` (mode 600). Profiles are ungrouped or in the groups `user` and `admin`. - **Cache storage**: any folder `cloudpathlib` understands; local folders and `s3://` are documented (credentials for S3 come from the standard AWS configuration), `gs://` and `az://` are accepted by the setting. ## Order Build the state layer first (`auth-profiles`, `config-cache`, the cache layout of `cache-keys`), then `flatten` and `workspace-entries`, then the saved-file commands and `cached-reads`, then the API commands one by one (workspaces list, apps, users, reports) and last the scans (`scan-batch`, `scan get`). A build that has the offline commands and the cache already serves scripted users who work from saved files. ## assumptions/api-limits.toml ```toml id = "K-api-limits" statement = "The admin endpoints are throttled by Power BI and the tool does not back off beyond a fixed retry on HTTP 500, 502 and 504; users keep to the limits with `--wait-interval`, scan `interval`/`timeout` and by scanning workspaces one by one." basis = "Retries are fixed (5, backoff factor 0.3); there is no handling of HTTP 429 or Retry-After. The scan batch and report-users commands expose waiting options for this reason." monitor = "HTTP 429 answers." ``` ## assumptions/api-shape.toml ```toml id = "K-api-shape" statement = "The Power BI REST API answers with the shapes the tool reads: lists under `value`, failures under an `error` key, user access under `ArtifactAccessEntities`, scan ids under `id`, scan states `NotStarted`, `Running`, `Succeeded` or `Failed`, and HTTP 202 while a scan result is not ready." basis = "These are the documented responses of the endpoints used (admin groups, apps, reports, report users, user artifact access, workspace info scans). The code reads them without checking." monitor = "none" ``` ## assumptions/bearer-token.toml ```toml id = "K-bearer-token" statement = "The user supplies a valid Power BI bearer token (for the admin commands, one with admin rights on the tenant) and replaces it when it expires; the tool never obtains, refreshes or validates one." basis = "No sign-in code exists; `pbi auth` only stores the string it is given. Power BI tokens are short-lived, so long scripts see HTTP 401 once the token has expired." monitor = "HTTP 401 and 403 answers from the API; the tool itself reports the failure." ``` ## assumptions/keyring-optional.toml ```toml id = "K-keyring-optional" statement = "A machine may have no OS keyring. Then tokens are kept in `~/.pbi_cli/credentials.json`; with a keyring they are kept there (service `pbi-cli`, user name = profile name) and the file is not used. The checks run without a keyring." basis = "The code probes the keyring and falls back to the file on any keyring error, including no backend and, on Windows, tokens too long for the credential manager." monitor = "`pbi profile list` shows ✓ for a profile only if its token is found in either place." ``` ## assumptions/single-user.toml ```toml id = "K-single-user" statement = "One `pbi` process at a time uses a home directory and a cache folder, and the machine's clock does not go backwards between runs." basis = "`config.yaml`, `credentials.json` and cache files are rewritten without locking. Cache versions are local timestamps to the second, so two saves of one key within a second overwrite each other and a clock change can reorder versions." monitor = "none" ``` ## obligations.toml ```toml # Obligations of the emptymalei/powerbi-cli spec pack (format: docs/PACK_FORMAT.md). # # Each obligation is one decision a builder could get wrong, with the checks # that pin it down. status: decided (required), default (chosen; a fork may # change it with a note), open (deliberately undecided; builds may differ). # Checks are "#example:" or # "#". A required obligation no check can # express yet has checks = [] and unchecked = "". [[obligation]] id = "auth.token-prefix" statement = "A token given as `Bearer ` is stored without the prefix." status = "decided" checks = [ "auth-profiles#example:2", "auth-profiles#test_auth_saves_the_token_and_activates_the_profile_when_none_is" ] [[obligation]] id = "auth.activation" statement = "Without `-g` a saved profile becomes the ungrouped active profile when none is active, or when it is named `default`; with `-g` it becomes the group's active profile when the group has none. An existing active profile otherwise stays, and a grouped save leaves the ungrouped profiles alone." status = "decided" checks = [ "auth-profiles#example:1", "auth-profiles#example:3", "auth-profiles#example:5", "auth-profiles#example:6", "auth-profiles#example:7", "auth-profiles#test_auth_saves_the_token_and_activates_the_profile_when_none_is" ] [[obligation]] id = "auth.token-per-name" statement = "Tokens are kept once per profile name in `~/.pbi_cli/credentials.json` (a JSON object, indent 2), shared by the ungrouped and grouped profiles of that name: saving replaces the token, deleting a profile deletes it, even if another list still has a profile of that name." status = "decided" assumes = ["K-keyring-optional"] checks = [ "auth-profiles#example:1", "auth-profiles#example:8", "auth-profiles#example:26", "auth-profiles#test_auth_saves_the_token_and_activates_the_profile_when_none_is", "auth-profiles#test_delete_removes_the_profile_and_its_token_and_promotes_the_first_remaining" ] [[obligation]] id = "auth.credentials-private" statement = "The credentials file is readable and writable only by its owner (mode 600)." status = "decided" kind = "security" assumes = ["K-keyring-optional"] checks = [] unchecked = "A check that reports the permission bits of the files a command wrote; the harness reports only their content." [[obligation]] id = "auth.keyring" statement = "When an OS keyring works, tokens are stored there (service `pbi-cli`, user name = profile name) and read from there first; the credentials file is only the fallback, also when the keyring raises any error." status = "decided" kind = "security" assumes = ["K-keyring-optional"] checks = [] unchecked = "A run against a fake keyring backend, which the harness does not provide." [[obligation]] id = "auth.legacy-import" statement = "Old installations keep working: a legacy `profiles.json` is imported into a new `config.yaml`, and a legacy `auth.json` token becomes the ungrouped profile `default` when there are no ungrouped profiles." status = "default" checks = [ "auth-profiles#example:9", "auth-profiles#example:12" ] [[obligation]] id = "auth.resolution" statement = "A command uses the active profile of its group (`user`, or `admin` for the admin-only commands) if that is one of the group's profiles; otherwise the ungrouped active profile, which must be one of the ungrouped profiles; it then needs a token for it. A group profile without a token does not fall back. Failures exit 1 with an `Error:` line naming the group or profile." status = "decided" checks = [ "auth-profiles#example:29", "auth-profiles#example:30", "auth-profiles#example:31", "auth-profiles#example:32", "auth-profiles#example:33", "auth-profiles#example:34", "auth-profiles#example:35", "auth-profiles#example:36", "auth-profiles#example:37", "auth-profiles#example:38", "auth-profiles#example:39", "auth-profiles#test_user_commands_take_the_user_groups_profile_else_the_ungrouped_one", "auth-profiles#test_admin_commands_fail_the_same_way_naming_the_admin_group" ] [[obligation]] id = "profile.list" statement = "`profile list` shows the ungrouped profiles (✓ when a token is stored under the name, ✗ when not, `(active)` on the active one) and then each group, with a hint for each empty part." status = "decided" checks = [ "auth-profiles#example:10", "auth-profiles#example:11", "auth-profiles#example:12", "auth-profiles#test_list_shows_ungrouped_then_each_group_with_token_marks" ] [[obligation]] id = "profile.switch" statement = "`profile switch NAME [-g G]` makes NAME the active profile of that list; an unknown NAME, or an empty list, is reported and changes nothing, with exit 0." status = "decided" checks = [ "auth-profiles#example:13", "auth-profiles#example:14", "auth-profiles#example:15", "auth-profiles#example:16", "auth-profiles#example:17", "auth-profiles#example:18", "auth-profiles#test_switch_changes_only_the_active_profile" ] [[obligation]] id = "profile.switch-interactive" statement = "Without a name, `profile switch` lists the profiles numbered from 1, reads the number from standard input (an empty answer is 1) and switches; a number out of range prints `Invalid selection` and changes nothing." status = "default" checks = [ "auth-profiles#example:19", "auth-profiles#example:20", "auth-profiles#example:21" ] [[obligation]] id = "profile.delete" statement = "`profile delete NAME [-g G]` asks for confirmation (`--yes` skips it; declining aborts with exit 1), removes the profile and its token, and when it was the active one activates the first remaining profile of that list (none if none remain), announcing it. An unknown name is reported with exit 0." status = "decided" checks = [ "auth-profiles#example:22", "auth-profiles#example:23", "auth-profiles#example:24", "auth-profiles#example:25", "auth-profiles#example:26", "auth-profiles#example:27", "auth-profiles#example:28", "auth-profiles#test_delete_removes_the_profile_and_its_token_and_promotes_the_first_remaining" ] [[obligation]] id = "config.file" statement = "`~/.pbi_cli/config.yaml` is a block-style YAML mapping with `active_profile`, `profiles`, `groups` (`user`, `admin`, each with `active_profile` and `profiles`), `default_output_folder`, `cache_folder` and `cache_enabled` (default true). A missing or unreadable file reads as the defaults; a change rewrites the whole file and keeps all other keys as they were." status = "decided" checks = [ "auth-profiles#example:1", "auth-profiles#example:4", "config-cache#example:5", "config-cache#example:13", "config-cache#example:14", "config-cache#test_enable_and_disable_change_only_cache_enabled", "config-cache#test_set_folder_stores_one_absolute_path_and_keeps_the_other_keys" ] [[obligation]] id = "config.set-folder" statement = "`config set-output-folder P` and `set-cache-folder P` drop surrounding quotes from P, expand `~` to the home directory, make a relative path absolute from the current directory and store it without a trailing slash; they print `✓ ... set to: ` and a hint line." status = "decided" checks = [ "config-cache#example:3", "config-cache#example:4", "config-cache#example:5", "config-cache#example:8", "config-cache#test_set_folder_stores_one_absolute_path_and_keeps_the_other_keys" ] [[obligation]] id = "config.cloud-cache-folder" statement = "A cache folder starting with `s3://`, `gs://` or `az://` is stored and shown as given." status = "decided" checks = [ "config-cache#example:9", "config-cache#example:10", "config-cache#test_set_cache_folder_keeps_cloud_locations" ] [[obligation]] id = "config.get-and-switch" statement = "`get-output-folder` and `get-cache-folder` print the stored folder (the cache one also `Cache enabled: True|False`) or say it is not configured; `enable-cache` and `disable-cache` set `cache_enabled` and print `✓ Cache enabled` / `✓ Cache disabled`." status = "decided" checks = [ "config-cache#example:6", "config-cache#example:7", "config-cache#example:11", "config-cache#example:12", "config-cache#example:13", "config-cache#example:14", "config-cache#test_enable_and_disable_change_only_cache_enabled" ] [[obligation]] id = "config.show" statement = "`config show` prints the active ungrouped profile, default output folder, cache folder, whether the cache is enabled, the profile count, the ungrouped profiles, and per group its profile count and active profile." status = "default" checks = [ "config-cache#example:1", "config-cache#example:2", "config-cache#test_show_lists_the_settings" ] [[obligation]] id = "cache.layout" statement = "The cache folder holds `//.json`. A version is a directory under the key that contains `.json`; versions are listed newest first (descending string order); a key is listed when it has at least one version, keys sorted; the latest data of a key is its first version." status = "decided" checks = [ "cache-keys#example:9", "cache-keys#example:10", "cache-keys#example:11", "cache-keys#test_index_lists_versions_newest_first", "config-cache#example:17", "config-cache#example:19", "config-cache#example:20", "config-cache#test_list_counts_versions_per_key_or_lists_the_versions_of_one_key" ] [[obligation]] id = "cache.entry-format" statement = "A save creates one file `//.json` whose version is the local time as `YYYYMMDD_HHMMSS`, holding a JSON object with `cache_key`, `cached_at` (ISO 8601 time), `version`, `metadata` and `data`." status = "decided" assumes = ["K-single-user"] checks = [ "cache-keys#example:7", "cache-keys#example:8", "cache-keys#test_entry_is_one_versioned_file" ] [[obligation]] id = "cache.keys" statement = "The cache key of `workspaces list` is `workspaces`, of `apps list` `apps_`, of `users user-access` `user_access_`; the request metadata saved is `{top, expand, filter}`, `{role}` and `{user_id}`." status = "decided" checks = [ "cache-keys#example:1", "cache-keys#example:2", "cache-keys#example:3", "cache-keys#example:4", "cache-keys#example:5", "cache-keys#example:6", "cache-keys#test_workspaces_key_and_metadata", "cache-keys#test_apps_key_has_the_role", "cache-keys#test_user_key_keeps_an_id_that_is_already_a_slug", "cache-keys#test_user_key_slugs_other_ids", "cached-reads#example:10", "cached-reads#example:11", "cached-reads#example:12", "cached-reads#example:13", "cached-reads#test_apps_read_the_key_of_their_role", "cached-reads#test_user_access_is_cached_under_the_slug_of_the_id" ] [[obligation]] id = "cache.list-and-clear" statement = "`cache list [-k KEY]` and `cache clear [-k KEY] [-v VERSION]` print what is in the cache or report what was cleared, as in the examples; `clear` asks for confirmation (`--yes` skips; declining aborts with exit 1); `-v` without `-k` is refused with a message (exit 0); with no cache folder both say so." status = "decided" checks = [ "config-cache#example:15", "config-cache#example:16", "config-cache#example:17", "config-cache#example:18", "config-cache#example:19", "config-cache#example:20", "config-cache#example:21", "config-cache#example:22", "config-cache#example:23", "config-cache#example:24", "config-cache#example:25", "config-cache#example:26", "config-cache#example:27", "config-cache#test_list_counts_versions_per_key_or_lists_the_versions_of_one_key", "config-cache#test_clear_reports_what_it_cleared" ] [[obligation]] id = "cache.clear-effect" statement = "`cache clear` removes everything in the folder (the folder stays), the key's whole folder, or the one version's file, and reports success even when nothing was there." status = "decided" assumes = ["K-single-user"] checks = [] unchecked = "A check that sees deleted files; the harness reports only files created or changed." [[obligation]] id = "cache.read-conditions" statement = "A list command reads the cache only with `--use-cache` or `--cache-only`, and only when a cache folder is set and caching is enabled." status = "decided" checks = [ "cached-reads#example:3", "cached-reads#example:4", "cached-reads#example:5", "cached-reads#example:6", "cached-reads#example:7", "cached-reads#test_cache_only_without_a_usable_cache_fails_with_exit_1", "cached-reads#test_use_cache_that_misses_goes_on_to_the_api_and_needs_a_token" ] [[obligation]] id = "cache.read-latest" statement = "A read takes the latest version of the command's key." status = "decided" checks = [ "cached-reads#example:1", "cached-reads#test_cache_only_shows_the_latest_version_of_the_key" ] [[obligation]] id = "cache.notice-from-file" statement = "On a hit the first line is `Using cached data from (version: )` with both fields taken from inside the cache file, not from the folder name." status = "default" checks = [ "cached-reads#example:1", "cached-reads#example:13", "cached-reads#test_cache_only_shows_the_latest_version_of_the_key" ] [[obligation]] id = "cache.unreadable-latest" statement = "If the latest version's file cannot be read as JSON the cache counts as unavailable; an older version is not used instead." status = "default" checks = [ "cached-reads#example:2", "cached-reads#test_cache_only_without_a_usable_cache_fails_with_exit_1" ] [[obligation]] id = "cache.miss-cache-only" statement = "With `--cache-only` and no usable cache, standard output says `Error: Cache not available and --cache-only was specified` (or, when no cache is set up or it is disabled, `Error: Cache not configured and --cache-only was specified` and a hint line), and the command exits 1 with `Aborted!`." status = "decided" checks = [ "cached-reads#example:2", "cached-reads#example:3", "cached-reads#example:4", "cached-reads#example:5", "cached-reads#example:14", "cached-reads#test_cache_only_without_a_usable_cache_fails_with_exit_1" ] [[obligation]] id = "cache.miss-use-cache" statement = "With only `--use-cache` and an empty cache the command says `Cache not available, fetching from API...` (not when no cache is set up) and goes on to the API." status = "decided" checks = [ "cached-reads#example:6", "cached-reads#example:7", "cached-reads#test_use_cache_that_misses_goes_on_to_the_api_and_needs_a_token" ] [[obligation]] id = "cache.table" statement = "Without a target folder the result is printed as a blank line, a rule of 80 `=`, the title (`Workspaces: n record(s)`, `Apps (): n record(s)`, `User Access Information for: `), a rule, the table (header line, one line per record) and a rule; with no records it prints `No workspaces found.` / `No apps () found.`." status = "decided" checks = [ "cached-reads#example:1", "cached-reads#example:8", "cached-reads#example:10", "cached-reads#example:11", "cached-reads#example:13", "cached-reads#test_cache_only_shows_the_latest_version_of_the_key", "cached-reads#test_apps_read_the_key_of_their_role", "cached-reads#test_user_access_is_cached_under_the_slug_of_the_id" ] [[obligation]] id = "cache.table-columns" statement = "The workspaces table shows `id, name, type, state, isReadOnly, isOnDedicatedCapacity` and the apps table `id, name, description, publishedBy, lastUpdate`, in that order and only those that exist; when none exists all columns are shown, nested objects flattened into dotted columns." status = "default" checks = [ "cached-reads#example:1", "cached-reads#example:9", "cached-reads#example:10", "cached-reads#test_cache_only_shows_the_latest_version_of_the_key", "cached-reads#test_apps_read_the_key_of_their_role" ] [[obligation]] id = "cache.save-files" statement = "With `--target-folder`, the list commands save `.json` (the data on one line) and, with `-ft excel`, `.xlsx`; `-n` sets the name (default `workspaces`, `apps`); the folder is created when missing and announced with `creating folder `." status = "decided" checks = [ "cached-reads#example:15", "cached-reads#example:16", "cached-reads#test_target_folder_gets_the_cached_data_as_one_line_json" ] [[obligation]] id = "cache.target-under-default-output" statement = "A relative target folder is placed under `default_output_folder` when one is set, else under the current directory." status = "decided" checks = [ "cached-reads#example:16", "cached-reads#test_target_folder_gets_the_cached_data_as_one_line_json" ] [[obligation]] id = "cache.user-access-save" statement = "`users user-access` with `--target-folder` is meant to save `.json` / `.xlsx` like the other list commands; the original only prints and saves nothing." status = "open" checks = [] [[obligation]] id = "cache.workspaces-excel-from-cache" statement = "`workspaces list --use-cache|--cache-only -ft excel -tf ...`: what is saved as the workbook when the data came from the cache (the original crashes after writing the JSON)." status = "open" checks = [] [[obligation]] id = "flatten.workspaces" statement = "Workspaces become seven sheets, in the order users, reports, dashboards, datasets, dataflows, workbooks, workspace: one row per child with the workspace's own (non-list) fields, in order, followed by the child's fields as `_`; sheet `workspace` has one row of own fields per workspace. Other list fields are dropped; sheets with no rows are kept." status = "decided" checks = [ "flatten#example:1", "flatten#example:3", "flatten#test_workspaces_one_row_per_child_repeating_the_workspace_fields" ] [[obligation]] id = "flatten.app" statement = "An app becomes the sheets users, reports, dashboards and app by the same rules." status = "decided" checks = [ "flatten#example:4", "flatten#example:5", "flatten#test_app_one_row_per_child_repeating_the_app_fields" ] [[obligation]] id = "flatten.report-users" statement = "Report users are flattened per workspace name: one row per user of each report, with the report's fields (without the users field) and the user's fields as `_` (key `reports_users_value` unless given); a report without users produces no row." status = "decided" checks = [ "flatten#example:6", "flatten#example:7", "flatten#test_report_users_one_row_per_user_of_each_report", "flatten#test_report_users_custom_key" ] [[obligation]] id = "flatten.child-wins" statement = "When a child's prefixed field has the same name as an own field, the child's value replaces it in place." status = "default" checks = [ "flatten#example:2" ] [[obligation]] id = "flatten.workbook" statement = "Each sheet is written as one worksheet of the same name, with a header row of the column names, no index column, and nested objects in a cell spread over dotted columns (`a.b`). (`apps list -ft excel` is an exception in the original: it has an index column; see REVIEW.)" status = "decided" checks = [] unchecked = "A check that reads the worksheets of an .xlsx file; the harness reports only that a file starts like a workbook." [[obligation]] id = "flatten.empty-listing" statement = "Converting or saving as Excel a listing with no workspaces (an empty `value`, or no `value`): the original fails with an internal error." status = "open" checks = [] [[obligation]] id = "convert.workbook" statement = "`workspaces format-convert -s S -t T` writes the workbook of the `flatten` sheets to T, whatever T is called, without needing a token; a missing or non-JSON S, or a T in a missing folder, fails." status = "decided" checks = [ "file-converters#example:1", "file-converters#example:2", "file-converters#example:3", "file-converters#example:4", "file-converters#example:5", "file-converters#test_format_convert_writes_a_workbook_under_the_given_name", "file-converters#test_format_convert_fails_on_unreadable_source" ] [[obligation]] id = "convert.apps-token" statement = "`apps augment` and `reports users` resolve the user group's token before anything else and write nothing without one." status = "decided" checks = [ "file-converters#example:8", "file-converters#example:15", "file-converters#test_without_a_token_nothing_is_written" ] [[obligation]] id = "convert.apps-no-apps" statement = "A source with no apps (no `value`, or an empty list) needs no request: a json target holds `[]`." status = "decided" checks = [ "file-converters#example:6", "file-converters#example:7", "file-converters#example:13", "file-converters#example:14", "file-converters#test_json_target_of_no_apps_is_an_empty_list" ] [[obligation]] id = "convert.apps-source" statement = "A missing source, or a source that is not a JSON object, fails." status = "decided" checks = [ "file-converters#example:9", "file-converters#example:12", "file-converters#example:16", "file-converters#example:19", "file-converters#test_a_source_that_is_not_an_object_fails" ] [[obligation]] id = "convert.excel-target" statement = "With `-ft excel` the target is a folder without a file extension, created when missing; a target with an extension is refused and nothing is written (the original fails with an internal error where it means a usage error)." status = "default" checks = [ "file-converters#example:10", "file-converters#example:11", "file-converters#example:17", "file-converters#example:18", "file-converters#test_excel_target_is_a_folder_without_extension" ] [[obligation]] id = "convert.apps-fetch" statement = "With apps in the source each app is fetched in full (`app.endpoint-app`); one the API answers with an error is reported in red (`Cannot download `) and skipped; a json target holds the list of fetched apps, an excel target gets `_.xlsx` per app (sheets of `flatten` kind app)." status = "decided" checks = [] unchecked = "A fake API behind the base URL, which is hard-coded." [[obligation]] id = "convert.report-users" statement = "`reports users` also fetches the users of each report of each fetched app (`app.endpoint-report-users`) and replaces the report by the users answer's fields followed by the report's fields; a report whose fetch fails with an error answer is dropped from the app and logged; an excel target gets `__report_users.xlsx`." status = "decided" checks = [] unchecked = "A fake API behind the base URL, which is hard-coded." [[obligation]] id = "scan.entry-shape" statement = "Each `workspace_ids` entry is a string (an id) or a mapping with a non-empty `id` and an optional `name`; entries keep their order, extra keys are ignored." status = "decided" checks = [ "workspace-entries#example:1", "workspace-entries#example:4", "workspace-entries#example:5", "workspace-entries#test_entries_without_a_name_use_the_id", "workspace-entries#test_order_is_kept_and_extra_keys_are_ignored" ] [[obligation]] id = "scan.label-and-file-name" statement = "An entry's label is ` ()` or the id; its result file is `-.json`, or `.json` when there is no name or its slug is empty. The slug is ASCII (accents folded), lowercase letters and digits with every other run of characters as one `-`, no leading or trailing `-`." status = "decided" checks = [ "workspace-entries#example:1", "workspace-entries#example:2", "workspace-entries#example:3", "workspace-entries#example:4", "workspace-entries#test_a_name_is_slugged_into_the_file_name" ] [[obligation]] id = "scan.entry-validation" statement = "`workspace_ids` that is not a list, or an entry that is not a string or a mapping with an `id`, makes the config invalid." status = "decided" checks = [ "workspace-entries#example:6", "workspace-entries#example:7", "workspace-entries#example:8", "workspace-entries#example:9", "workspace-entries#example:10", "workspace-entries#test_workspace_ids_must_be_a_list", "workspace-entries#test_one_bad_entry_makes_the_input_invalid", "scan-batch#example:10", "scan-batch#example:11", "scan-batch#example:12", "scan-batch#test_config_problems_come_before_any_setting_is_looked_at" ] [[obligation]] id = "scan.config-order" statement = "Config problems are reported one at a time, in this order: not a mapping; bad `workspace_ids`; no entries; no `target_folder`; then (after the folder is created) a non-boolean flag in the order lineage, datasource_details, dataset_schema, dataset_expressions, get_artifact_users; then non-numeric, then non-positive `interval`/`timeout`. Each is an `Error:` line and exit 1." status = "decided" checks = [ "scan-batch#example:6", "scan-batch#example:7", "scan-batch#example:8", "scan-batch#example:9", "scan-batch#example:10", "scan-batch#example:11", "scan-batch#example:12", "scan-batch#example:13", "scan-batch#example:14", "scan-batch#example:15", "scan-batch#example:17", "scan-batch#example:23", "scan-batch#test_config_problems_come_before_any_setting_is_looked_at", "scan-batch#test_settings_are_checked_flags_first_then_numbers" ] [[obligation]] id = "scan.flags-boolean" statement = "A flag must be a YAML boolean; `1`, a string and null are refused with `'' must be a boolean value.`" status = "decided" checks = [ "scan-batch#example:16", "scan-batch#example:17", "scan-batch#example:18", "scan-batch#test_settings_are_checked_flags_first_then_numbers" ] [[obligation]] id = "scan.interval-timeout" statement = "`interval` (default 5) and `timeout` (default 300) are numbers or numeric strings; null or absent means the default; they must be above 0." status = "decided" checks = [ "scan-batch#example:4", "scan-batch#example:19", "scan-batch#example:20", "scan-batch#example:21", "scan-batch#example:22", "scan-batch#test_settings_are_checked_flags_first_then_numbers" ] [[obligation]] id = "scan.yaml-1-1" statement = "The config is read as YAML 1.1: `yes`, `no`, `on` and `off` are booleans." status = "default" checks = [ "scan-batch#example:3" ] [[obligation]] id = "scan.target-folder" statement = "`target_folder` is resolved like other output folders (relative: under `default_output_folder` when set, else the current directory), created when missing and announced as `creating folder ` before the flags are checked." status = "decided" checks = [ "scan-batch#example:1", "scan-batch#example:2", "scan-batch#example:16", "scan-batch#test_settings_are_checked_flags_first_then_numbers" ] [[obligation]] id = "scan.loop" statement = "Workspaces run one after another, each announced by a blank line and `=== Workspace