Popequer

Popequer is a way to organize your digital life around a text-based, entry-based, personal knownledge management system.

Built to be flexible, one of the initial goal is to have a way to edit and contribute to wikidata.org using text syntax.

But the goal is to be able to use it later as a collaborative knowledge management system to eventually replace wikibase.

This project is currently at the very begining with only the parser for now.

The popequer project currently have a unique mono-repo.

Components

• The popequer data entry language parser (PDEL)
  • Parse an entry to an intermediate Entry representation (IER)
• The Database/Indexer:
  • Directory crawler, markdown parser and indexer
  • Convertion from IER to a rich, validated and linked database
  • The query interpretor (that use an adaptation of the PDEL for the query language)
  • Indexer CLI
  • Query CLI
• User Front-end
  • Calendar view
  • Events/History view (Activity view)
  • Adaptative view (for any kind of relational and general database)
• Convertion utils from and to common formats (CSV, JSON, etc)
• Import/Export interface for specific uses
  • CalDAV interface
  • CardDAV interface
  • YouTube interface
  • E-Mail interface (IMAP)
  • File system interface
  • Any kind of interface to foreign software to pull activity data
• Popequer notebook day-to-day CLI helper
• The wikidata interface:
  • The wikidata to PDEL translator lib
  • The parsed PDEL to wikidata translator lib
  • The normal wikidata client
  • The wikidata CLI editor that use the libs

External components:

• Android app
• Desktop visualizer

Quick reference

To only run one test with stdout print not hidden:

cargo test parse::test_parse::test_parse_entry_claim -- --nocapture --exact

Test coverage

Today, source based coverage in Rust is the recommended way to do coverage. It’s also the most usable, most accurate, and most portable way to do coverage.

We use cargo-llvm-cov to analyze the test coverage of the popequer source code.

Instructions to install cargo-llvm-cov

Run cargo llvm-cov --open to compute and see the coverage.

The PDEL Parser

We will introduce a new markup language specification or format The purpose is to ease and simplify the creation, edition and management of wikibase-like items This boils down to being able to quickly make claims using human usable identifier or key Instead of remembering for each wikibase property an Id, we introduce a mapping between the key and the property id We will also introduce simple syntaxic sugar to prevent redundancy in our syntax It's a simpler form of the notation3 syntax which aims to a readable RDF syntax We aims to keep as close as possible to the RDF models but with specificities of wikibase in mind To put it in a nutshell, we want to create a language to edit wikibase with the same ease and practicality of the wikibase web interface.

The popequer database and data model

Compatible with the wikibase, the data model is relatively simple and allow to easily model any kind of knowledge.

The database must be design to allow for graph-like query search. Like: what are the friends of my friend? We could use neo4j but it's not very embededd. We could study what makes neo4j interesting as a graph database.

Ideally I would like to use an embeded graph-database

All datetime are UTC only (at least internally).

Flow and different representations of entries at any time

• PDEL raw text
• Preprocessed PDEL code string with offset table (shotcut expansion, comments removal) by the "preprocessor"
• Parsed PDEL AST (intermediate) outputed by the "pdel_parser"
• Pure entries extracted from the AST by the "parse_extractor" (FIXME: find a better name)
• Linked, resolved pure entries with properties
  • the final phase, is the name resolution
  • for each unresolved reference, try to check if it's a common wikidata name (defined in the items overloading), or else try to find it in others notebooks?
  • FIXME: find a way to still resolve the names of the entries with no explicit names, but with names created from linked entries
  • output a real graph-like structure

Definition of the MVP

There are two versions of the MVP

MVP for personal use with neovim

• Rust? treesitter extension that use the same codebase would be great
• Rust LSP server on a notebook
  • Implement LSP "GoTo" feature to resolve references
  • Implement LSP
  • Provide completion of properties AND items name
• Neovim plugin to add all the useful stuff
• Basic CLI util
  • launch an indexation and update of the DB
  • get the status

MVP for wikidata use with neovim

Wikidata editor

Getting started as a user

use the indexer

cargo run --bin indexer -- -s ./examples/notebook/ -i ./.tmp/notebook_internals/

Feature: generation of id after first parse

• scenario 1: we don't handle it, and let the user add it
• scenario 2: we first parse, then we add it to the raw code, then we write it back

we need to have a function signature that working independant of file handle we need to take in a StringIO as output same : a StringIO

so we have a function to work with a code only we need to iterate after

Tests

for now the convention for order in params when calling assert_eq is: (actual, expected)