Skip to content
YYUUGGOOPublic

About

A python based text mining toolkit mainly for japanese language pdfs

Topics

Resources

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

10 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

QWIK

QWIK is a local desktop application for searching and analyzing text in PDF and TXT sources. It provides a visual keyword-in-context (KWIC) workflow with PDF page previews and highlights, document metadata, tokenization options, frequency and co-occurrence views, BibTeX matching, coding tools, and locally saved project databases.

The project is mainly aimed at Japanese and European-language texts. Japanese analysis uses fugashi/IPADIC; the multilingual POS path uses the Stanford POS Tagger for English, Arabic, Chinese, French, German, and Spanish.

QWIK supports macOS and Windows. It uses Python’s Tkinter desktop UI and PyMuPDF for PDF access. It is a desktop GUI, not a hosted service.

Default installation and features

The normal installation includes the core QWIK GUI and these runtime dependencies: PyMuPDF and fugashi/ipadic. The Stanford POS Tagger is a separate Java runtime bundle downloaded from its Stanford POS Models window.

The default GUI provides:

  • PDF and TXT source loading, folder loading, indexing, and PDF page previews.
  • KWIC search with Literal, Regex, Normalized, and Lemma modes.
  • Character, token, or sentence context, with page as the default analysis unit.
  • A fugashi/IPADIC tokenizer for Japanese morphological analysis.
  • Stanford POS Tagger integration for English, Arabic, Chinese, French, German, and Spanish.
  • Metadata editing, BibTeX loading/matching, codebooks, variants, join words, and exclude-word lists.
  • Term-frequency, collocate, subcorpus-keyword, co-occurrence, and analysis-dashboard views.
  • CSV and GraphML exports, plus local project save/reopen.

Semantic embeddings and semantic search are experimental. They are not installed by default; install the optional semantic extra only to evaluate them:

uv sync --extra semantic

The Stanford POS Tagger requires Java 8 or newer. Use the Stanford POS Models button inside QWIK to download its approximately 75 MB model bundle. The application keeps a local Java tagger server alive and reuses it across text units, so the model is loaded only once. Japanese projects should continue using fugashi/IPADIC. Stanford’s current bundle provides trained models for English, Arabic, Chinese, French, German, and Spanish; other languages remain unsupported unless a compatible model is supplied.

Ollama translation is also experimental. It requires a local Ollama installation, the qwen2.5:14b model (or the model configured by the application), and a locally reachable Ollama server at http://127.0.0.1:11434. Selected text is sent only to that local endpoint. QWIK does not provide or install Ollama models.

Requirements

  • macOS or Windows.
  • Python 3.10 or newer.
  • Tkinter. Official Python installers for macOS and Windows normally include it. If import tkinter fails, install or repair Python with Tcl/Tk support before installing QWIK.

Linux is not a supported desktop target in this release. The automated checks run on Ubuntu and may not reproduce every macOS or Windows GUI behavior.

Install

QWIK uses uv for environments, dependencies, and the locked development workflow. Download the source or clone the repo. Then from the repo directory:

# Use a Python build with Tcl/Tk support for the desktop GUI.
uv venv --python /Library/Frameworks/Python.framework/Versions/3.12/bin/python3
uv sync

On other systems, replace the interpreter path with a Tk-enabled Python 3.12 executable. Minimal Python builds without Tcl/Tk can run core tests but cannot launch the desktop window.

Install the development tools when running the test suite:

uv sync --extra dev
uv run pytest -q

Run the application through the managed environment:

uv run qwik

QWIK writes lifecycle and worker progress to the terminal. For detailed diagnostics, use either of these forms:

uv run qwik --verbose
QWIK_LOG_LEVEL=DEBUG uv run qwik

Run

After syncing QWIK through uv, launch it with:

uv run qwik

Use uv run qwik --verbose when diagnosing slow indexing, tokenization, or network analysis. The co-occurrence calculation and Stanford server startup run in background workers so the Tkinter window remains responsive.

On Windows PowerShell, the same command is:

uv run qwik

Use New Project or Open Project, add PDF/TXT sources, select Index Sources, and then use the Analysis tab. The command starts the Tkinter GUI; it does not start a web server.

TXT corpus structure

QWIK accepts KH Coder–style, XML-like heading markup in TXT corpora. This is heading markup rather than a strict XML schema: QWIK recognizes paired H1–H5 tags, case-insensitively, and treats the text between headings as content. Lower-level heading context is cleared when a higher-level heading starts.

For newspaper-style material, a typical hierarchy is year → month → article:

<H1>2025</H1>
<H2>dec</H2>
<H3>(現場へ!)全国方言サバイバル2:1 混ざり合うウチナーグチ</H3>
「遅れてるのかなあ」「めっちゃ……」。沖縄県宜野湾市のバス停にいると、
女子高校生たちの会話が聞こえた。

<H3>「鹿児島弁トランプ」、方言の魅力を再発見 児童文学作家が作製 /佐賀県</H3>
すべてのカードに鹿児島弁の言葉が書かれた「鹿児島弁トランプ」をつくった。

The sample is illustrative; headings and content can use different values and additional levels. When importing TXT files, QWIK lets you choose the date tag, title tag, and body tags. The defaults are H1 for date, H2 for title, and H2–H5 for body content. To treat each article in the hierarchy above as a document, select H1 as the date tag, H3 as the title tag, and include H3 (and any deeper levels) as body tags.

Local user data

Source PDFs and TXT corpora, project databases, BibTeX files, reports, CSV/GraphML exports, annotated PDFs, caches, downloaded model files, and other generated outputs are local user data. They may contain private or copyrighted research material and are intentionally excluded from the public repository by .gitignore. Keep backups of anything you need; ignored files are not versioned or published by this project.

Contributing and security

Please read the contribution guide before opening a pull request. Report suspected vulnerabilities using the security policy, not a public issue.

License

QWIK is released under the MIT License.

Acknowledgments

QWIK was inspired broadly by the text-mining tool KH Coder, including its hierarchical heading-marked text model and structured analysis workflows. The project came into existence partly because macOS support for KH Coder is commercial and its source code is closed.

A central aim is interoperability between QWIK and KH Coder. Where full interoperability is not yet possible, QWIK aims to remain compatible with KH Coder's conventions, especially its TXT corpus structuring and coding system. QWIK is an independent project and is not affiliated with KH Coder.

About

A python based text mining toolkit mainly for japanese language pdfs

Topics

Resources

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Contributors

Languages