diff --git a/README.md b/README.md index 0c2bb4ab..f8959572 100644 --- a/README.md +++ b/README.md @@ -65,7 +65,7 @@ Contributions welcome! See [contribution guidelines](https://docs.datajoint.com/ **Quick fixes:** Fork, edit markdown in `src/`, submit PR. -**Tutorial notebooks:** Re-execute after changes: +**Single notebook:** Re-execute one notebook in place: ```bash docker compose exec docs jupyter nbconvert --to notebook --execute --inplace \ /main/src/tutorials/YOUR_NOTEBOOK.ipynb @@ -94,6 +94,19 @@ banner doesn't match `extra.datajoint_version`: python scripts/check_notebook_versions.py ``` +Notes on the execution environment: + +- **Credentials.** The Compose databases use the password `tutorial` — + `root` for MySQL (`MYSQL_ROOT_PASSWORD`) and `postgres` for PostgreSQL (`POSTGRES_USER` / + `POSTGRES_PASSWORD`), both set in `docker-compose.yaml`. `MODE=EXECUTE*` supplies them to the + notebooks automatically; you only need them to connect to a Compose database from outside it. +- **Graphviz.** `dj.Diagram`'s notebook display calls the Graphviz `dot` binary, which the docs + image installs. This is why committed diagram outputs always render, and why the same cell + fails on a host without Graphviz — see + [Installation → Troubleshooting](https://docs.datajoint.com/how-to/installation/#djdiagram-raises-filenotfounderror). +- **DataJoint version.** `pip_requirements.txt` installs `datajoint@master`, so regenerated + outputs track the development branch rather than the last release. + ## Related - [datajoint-python](https://github.com/datajoint/datajoint-python) — Core library diff --git a/src/how-to/installation.md b/src/how-to/installation.md index 8b8980de..b3a78aa5 100644 --- a/src/how-to/installation.md +++ b/src/how-to/installation.md @@ -2,6 +2,13 @@ Install DataJoint Python and set up your environment. +## Requirements + +- Python 3.10+ +- MySQL 8.0.13+ or PostgreSQL 15+ +- Network access to database server +- Linux, macOS, or Windows (see [Platform support](#platform-support)) + ## Install DataJoint 2.0+ ```bash @@ -11,7 +18,7 @@ pip install datajoint **With optional dependencies:** ```bash -# For diagram visualization (matplotlib, ipython) +# For Diagram.draw(), which renders diagrams into a matplotlib figure pip install datajoint[viz] # For polars DataFrame support @@ -56,6 +63,10 @@ print(dj.__version__) DataJoint connects to either MySQL or PostgreSQL. MySQL has been supported since the original release; PostgreSQL support was added in **2.1**, and the `database.name` setting for selecting a non-default PostgreSQL database was added in **2.2.1**. See [Configure Database Connection](configure-database.md#postgresql-backend) for the full configuration reference. +### DataJoint.com (Recommended) + +[DataJoint.com](https://datajoint.com) provides fully managed infrastructure for scientific data pipelines—cloud or on-premises—with comprehensive support, automatic backups, object storage, and team collaboration features. + ### Local Development (Docker) ```bash @@ -76,10 +87,6 @@ docker run -d \ postgres:15 ``` -### DataJoint.com (Recommended) - -[DataJoint.com](https://datajoint.com) provides fully managed infrastructure for scientific data pipelines—cloud or on-premises—with comprehensive support, automatic backups, object storage, and team collaboration features. - ### Self-Managed Cloud Databases - **Amazon RDS** — MySQL, Aurora MySQL, PostgreSQL, or Aurora PostgreSQL @@ -88,12 +95,101 @@ docker run -d \ See [Configure Database Connection](configure-database.md) for connection setup. -## Requirements +## Your First Pipeline -- Python 3.10+ -- MySQL 8.0.13+ or PostgreSQL 15+ -- Network access to database server -- Linux, macOS, or Windows (see [Platform support](#platform-support)) +With DataJoint installed and a database reachable, this is the shortest end-to-end path: point +DataJoint at your database, declare a few tables, insert a row, and run a computation. + +Configure your project (see [Configuration](../reference/configuration.md) for the full +reference): + +```bash +# Non-sensitive settings +echo '{"database": {"host": "localhost", "port": 3306}}' > datajoint.json + +# Credentials, kept out of the project files +mkdir -p .secrets +echo "" > .secrets/database.user +echo "" > .secrets/database.password +chmod 600 .secrets/* +``` + +Define and populate a simple pipeline: + +```python +import datajoint as dj + +schema = dj.Schema('my_pipeline') + +@schema +class Subject(dj.Manual): + definition = """ + subject_id : int32 + --- + name : varchar(100) + date_of_birth : date + """ + +@schema +class Session(dj.Manual): + definition = """ + -> Subject + session_idx : int16 + --- + session_date : date + """ + +@schema +class SessionAnalysis(dj.Computed): + definition = """ + -> Session + --- + result : float64 + """ + + def make(self, key): + # Compute result for this session + self.insert1({**key, 'result': 42.0}) + +# Insert data +Subject.insert1({'subject_id': 1, 'name': 'M001', 'date_of_birth': '2026-01-15'}) +Session.insert1({'subject_id': 1, 'session_idx': 1, 'session_date': '2026-01-06'}) + +# Run computations +SessionAnalysis.populate() +``` + +`Subject` and `Session` are entered by hand; `SessionAnalysis` derives from `Session` and fills +itself when you call `populate()`. That dependency — declared with `->` — is the whole of the +[Relational Workflow Model](../explanation/relational-workflow-model.md) in miniature. + +## Running the Tutorial Notebooks + +The [tutorials](../tutorials/index.md) are published with their outputs, so you can read them +straight through. To run them yourself, add Jupyter to the environment above and get a copy of +the notebooks — each tutorial page has a download link, or clone the documentation repository +for all of them at once: + +```bash +pip install jupyterlab + +git clone https://github.com/datajoint/datajoint-docs.git +jupyter lab datajoint-docs/src/tutorials/ +``` + +Configuration works exactly as in the example above: DataJoint searches upward from the +notebook's directory for `datajoint.json` and reads credentials from the `.secrets/` directory +beside it. The repository ships a `datajoint.json` at its root pointing at `localhost:3306` — +edit it to match your own server, and add your own `.secrets/`. Each tutorial creates its own +schema, so they do not collide with each other or with your existing databases. + +Two dependencies are worth installing up front: + +- **Graphviz**, for the `dj.Diagram` cells — see + [Troubleshooting](#djdiagram-raises-filenotfounderror) below. +- **An object store**, for the tutorials that use ``, ``, or `` types. The + committed `datajoint.json` writes to a local directory, which needs no extra services; see + [Configure Storage](configure-storage.md) to point it elsewhere. ## Platform support @@ -140,3 +236,18 @@ GRANT ALL PRIVILEGES ON `your_schema%`.* TO 'username'@'%'; -- PostgreSQL GRANT ALL PRIVILEGES ON DATABASE my_db TO username; ``` + +### `dj.Diagram` raises `FileNotFoundError` + +Install Graphviz: + +```bash +brew install graphviz # macOS +sudo apt-get install graphviz # Debian/Ubuntu +conda install -c conda-forge graphviz # conda, any platform +``` + +Diagrams render through `pydot` — installed with DataJoint — which calls the Graphviz `dot` +executable. Graphviz is a system package, so `pip` cannot supply it. On Windows, install +from [graphviz.org/download](https://graphviz.org/download/) and put its `bin` directory on +your `PATH`. Verify with `dot -V`. diff --git a/src/index.md b/src/index.md index 524bf23d..8ddbc361 100644 --- a/src/index.md +++ b/src/index.md @@ -25,7 +25,7 @@ --- - Build your first pipeline with hands-on Jupyter notebooks + Learn DataJoint through complete, worked Jupyter-notebook pipelines [:octicons-arrow-right-24: Start learning](tutorials/index.md) @@ -65,4 +65,4 @@ --- -**New to DataJoint?** Start with the [:octicons-arrow-right-24: Quick Start tutorial](tutorials/index.md). +**New to DataJoint?** Visit [:octicons-arrow-right-24: Installation](how-to/installation.md) to install DataJoint and run your first pipeline, then work through the [:octicons-arrow-right-24: Tutorials](tutorials/index.md). diff --git a/src/reference/specs/diagram.md b/src/reference/specs/diagram.md index 0b00e75a..9d6a75e6 100644 --- a/src/reference/specs/diagram.md +++ b/src/reference/specs/diagram.md @@ -491,13 +491,18 @@ combined = dj.Diagram.from_sequence([schema1, schema2, schema3]) Operational methods (`cascade`, `restrict`, `counts`, `prune`) use `networkx`, which is always installed as a core dependency. -Diagram **visualization** requires optional dependencies: - -```bash -pip install matplotlib pygraphviz -``` - -If visualization dependencies are missing, `dj.Diagram` displays a warning and provides a stub class. Operational methods remain available regardless. +Diagram **visualization** additionally requires: + +- **Graphviz** — the `dot` executable. The default notebook display, `_repr_svg_()`, goes through + `make_svg()` → `make_dot()` → pydot → `dot`, so this is the path nearly all usage hits. `pydot` + ships with DataJoint, but Graphviz itself is a system package that `pip` cannot install + (`brew install graphviz`, `sudo apt-get install graphviz`). Without it, rendering raises + `FileNotFoundError` — see + [Installation → Troubleshooting](../../how-to/installation.md#djdiagram-raises-filenotfounderror). +- **matplotlib** — only for `Diagram.draw()`, a separate `make_image()` path that most usage never + touches: `pip install datajoint[viz]`. + +Operational methods remain available regardless. --- diff --git a/src/tutorials/index.md b/src/tutorials/index.md index c5f71695..50df89f9 100644 --- a/src/tutorials/index.md +++ b/src/tutorials/index.md @@ -2,76 +2,15 @@ Learn DataJoint by building real pipelines. -These tutorials guide you through building data pipelines step by step. Each tutorial -is a Jupyter notebook that you can run interactively. Start with the basics and -progress to domain-specific and advanced topics. +These tutorials guide you through building data pipelines step by step. Each tutorial is a +Jupyter notebook, published here with its code and its executed outputs, so you can follow the +whole pipeline without leaving the page. Start with the basics and progress to domain-specific +and advanced topics. -## Quick Start - -Install DataJoint: - -```bash -pip install datajoint -``` - -Configure database credentials in your project (see [Configuration](../reference/configuration.md)): - -```bash -# Create datajoint.json for non-sensitive settings -echo '{"database": {"host": "localhost", "port": 3306}}' > datajoint.json - -# Create secrets directory for credentials -mkdir -p .secrets -echo "root" > .secrets/database.user -echo "password" > .secrets/database.password -``` - -Define and populate a simple pipeline: - -```python -import datajoint as dj - -schema = dj.Schema('my_pipeline') - -@schema -class Subject(dj.Manual): - definition = """ - subject_id : int32 - --- - name : varchar(100) - date_of_birth : date - """ - -@schema -class Session(dj.Manual): - definition = """ - -> Subject - session_idx : int16 - --- - session_date : date - """ - -@schema -class SessionAnalysis(dj.Computed): - definition = """ - -> Session - --- - result : float64 - """ - - def make(self, key): - # Compute result for this session - self.insert1({**key, 'result': 42.0}) - -# Insert data -Subject.insert1({'subject_id': 1, 'name': 'M001', 'date_of_birth': '2026-01-15'}) -Session.insert1({'subject_id': 1, 'session_idx': 1, 'session_date': '2026-01-06'}) - -# Run computations -SessionAnalysis.populate() -``` - -Continue learning with the structured tutorials below. +**Want to run them yourself?** Every notebook can be downloaded and executed against your own +MySQL or PostgreSQL database. [Installation](../how-to/installation.md) covers the setup: +DataJoint, a database, Jupyter, and +[getting the notebooks](../how-to/installation.md#running-the-tutorial-notebooks). ## Learning Paths @@ -198,19 +137,3 @@ Extending DataJoint for specialized use cases: - [JSON Data Type](advanced/json-type.ipynb) — Semi-structured data in tables - [Distributed Computing](advanced/distributed.ipynb) — Multi-process and cluster workflows - [Custom Codecs](advanced/custom-codecs.ipynb) — Extending the type system - -## Running the Tutorials - -```bash -# Clone the repository -git clone https://github.com/datajoint/datajoint-docs.git -cd datajoint-docs - -# Start the tutorial environment -docker compose up -d - -# Launch Jupyter -jupyter lab src/tutorials/ -``` - -All tutorials use a local MySQL database that resets between sessions.