Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ Contributions welcome! See [contribution guidelines](https://docs.datajoint.com/

**Quick fixes:** Fork, edit markdown in `src/`, submit PR.

**Tutorial notebooks:** Re-execute after changes:
**Single notebook:** Re-execute one notebook in place:
```bash
docker compose exec docs jupyter nbconvert --to notebook --execute --inplace \
/main/src/tutorials/YOUR_NOTEBOOK.ipynb
Expand Down Expand Up @@ -94,6 +94,19 @@ banner doesn't match `extra.datajoint_version`:
python scripts/check_notebook_versions.py
```

Notes on the execution environment:

- **Credentials.** The Compose databases use the password `tutorial` —
`root` for MySQL (`MYSQL_ROOT_PASSWORD`) and `postgres` for PostgreSQL (`POSTGRES_USER` /
`POSTGRES_PASSWORD`), both set in `docker-compose.yaml`. `MODE=EXECUTE*` supplies them to the
notebooks automatically; you only need them to connect to a Compose database from outside it.
- **Graphviz.** `dj.Diagram`'s notebook display calls the Graphviz `dot` binary, which the docs
image installs. This is why committed diagram outputs always render, and why the same cell
fails on a host without Graphviz — see
[Installation → Troubleshooting](https://docs.datajoint.com/how-to/installation/#djdiagram-raises-filenotfounderror).
- **DataJoint version.** `pip_requirements.txt` installs `datajoint@master`, so regenerated
outputs track the development branch rather than the last release.

## Related

- [datajoint-python](https://github.com/datajoint/datajoint-python) — Core library
Expand Down
131 changes: 121 additions & 10 deletions src/how-to/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,13 @@

Install DataJoint Python and set up your environment.

## Requirements

- Python 3.10+
- MySQL 8.0.13+ or PostgreSQL 15+
- Network access to database server
- Linux, macOS, or Windows (see [Platform support](#platform-support))

## Install DataJoint 2.0+

```bash
Expand All @@ -11,7 +18,7 @@ pip install datajoint
**With optional dependencies:**

```bash
# For diagram visualization (matplotlib, ipython)
# For Diagram.draw(), which renders diagrams into a matplotlib figure
pip install datajoint[viz]

# For polars DataFrame support
Expand Down Expand Up @@ -56,6 +63,10 @@ print(dj.__version__)

DataJoint connects to either MySQL or PostgreSQL. MySQL has been supported since the original release; PostgreSQL support was added in **2.1**, and the `database.name` setting for selecting a non-default PostgreSQL database was added in **2.2.1**. See [Configure Database Connection](configure-database.md#postgresql-backend) for the full configuration reference.

### DataJoint.com (Recommended)

[DataJoint.com](https://datajoint.com) provides fully managed infrastructure for scientific data pipelines—cloud or on-premises—with comprehensive support, automatic backups, object storage, and team collaboration features.

### Local Development (Docker)

```bash
Expand All @@ -76,10 +87,6 @@ docker run -d \
postgres:15
```

### DataJoint.com (Recommended)

[DataJoint.com](https://datajoint.com) provides fully managed infrastructure for scientific data pipelines—cloud or on-premises—with comprehensive support, automatic backups, object storage, and team collaboration features.

### Self-Managed Cloud Databases

- **Amazon RDS** — MySQL, Aurora MySQL, PostgreSQL, or Aurora PostgreSQL
Expand All @@ -88,12 +95,101 @@ docker run -d \

See [Configure Database Connection](configure-database.md) for connection setup.

## Requirements
## Your First Pipeline

- Python 3.10+
- MySQL 8.0.13+ or PostgreSQL 15+
- Network access to database server
- Linux, macOS, or Windows (see [Platform support](#platform-support))
With DataJoint installed and a database reachable, this is the shortest end-to-end path: point
DataJoint at your database, declare a few tables, insert a row, and run a computation.

Configure your project (see [Configuration](../reference/configuration.md) for the full
reference):

```bash
# Non-sensitive settings
echo '{"database": {"host": "localhost", "port": 3306}}' > datajoint.json

# Credentials, kept out of the project files
mkdir -p .secrets
echo "<your-username>" > .secrets/database.user
echo "<your-password>" > .secrets/database.password
chmod 600 .secrets/*
```

Define and populate a simple pipeline:

```python
import datajoint as dj

schema = dj.Schema('my_pipeline')

@schema
class Subject(dj.Manual):
definition = """
subject_id : int32
---
name : varchar(100)
date_of_birth : date
"""

@schema
class Session(dj.Manual):
definition = """
-> Subject
session_idx : int16
---
session_date : date
"""

@schema
class SessionAnalysis(dj.Computed):
definition = """
-> Session
---
result : float64
"""

def make(self, key):
# Compute result for this session
self.insert1({**key, 'result': 42.0})

# Insert data
Subject.insert1({'subject_id': 1, 'name': 'M001', 'date_of_birth': '2026-01-15'})
Session.insert1({'subject_id': 1, 'session_idx': 1, 'session_date': '2026-01-06'})

# Run computations
SessionAnalysis.populate()
```

`Subject` and `Session` are entered by hand; `SessionAnalysis` derives from `Session` and fills
itself when you call `populate()`. That dependency — declared with `->` — is the whole of the
[Relational Workflow Model](../explanation/relational-workflow-model.md) in miniature.

## Running the Tutorial Notebooks

The [tutorials](../tutorials/index.md) are published with their outputs, so you can read them
straight through. To run them yourself, add Jupyter to the environment above and get a copy of
the notebooks — each tutorial page has a download link, or clone the documentation repository
for all of them at once:

```bash
pip install jupyterlab

git clone https://github.com/datajoint/datajoint-docs.git
jupyter lab datajoint-docs/src/tutorials/
```

Configuration works exactly as in the example above: DataJoint searches upward from the
notebook's directory for `datajoint.json` and reads credentials from the `.secrets/` directory
beside it. The repository ships a `datajoint.json` at its root pointing at `localhost:3306` —
edit it to match your own server, and add your own `.secrets/`. Each tutorial creates its own
schema, so they do not collide with each other or with your existing databases.

Two dependencies are worth installing up front:

- **Graphviz**, for the `dj.Diagram` cells — see
[Troubleshooting](#djdiagram-raises-filenotfounderror) below.
- **An object store**, for the tutorials that use `<blob@>`, `<npy@>`, or `<attach@>` types. The
committed `datajoint.json` writes to a local directory, which needs no extra services; see
[Configure Storage](configure-storage.md) to point it elsewhere.

## Platform support

Expand Down Expand Up @@ -140,3 +236,18 @@ GRANT ALL PRIVILEGES ON `your_schema%`.* TO 'username'@'%';
-- PostgreSQL
GRANT ALL PRIVILEGES ON DATABASE my_db TO username;
```

### `dj.Diagram` raises `FileNotFoundError`

Install Graphviz:

```bash
brew install graphviz # macOS
sudo apt-get install graphviz # Debian/Ubuntu
conda install -c conda-forge graphviz # conda, any platform
```

Diagrams render through `pydot` — installed with DataJoint — which calls the Graphviz `dot`
executable. Graphviz is a system package, so `pip` cannot supply it. On Windows, install
from [graphviz.org/download](https://graphviz.org/download/) and put its `bin` directory on
your `PATH`. Verify with `dot -V`.
4 changes: 2 additions & 2 deletions src/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@

---

Build your first pipeline with hands-on Jupyter notebooks
Learn DataJoint through complete, worked Jupyter-notebook pipelines

[:octicons-arrow-right-24: Start learning](tutorials/index.md)

Expand Down Expand Up @@ -65,4 +65,4 @@

---

**New to DataJoint?** Start with the [:octicons-arrow-right-24: Quick Start tutorial](tutorials/index.md).
**New to DataJoint?** Visit [:octicons-arrow-right-24: Installation](how-to/installation.md) to install DataJoint and run your first pipeline, then work through the [:octicons-arrow-right-24: Tutorials](tutorials/index.md).
19 changes: 12 additions & 7 deletions src/reference/specs/diagram.md
Original file line number Diff line number Diff line change
Expand Up @@ -491,13 +491,18 @@ combined = dj.Diagram.from_sequence([schema1, schema2, schema3])

Operational methods (`cascade`, `restrict`, `counts`, `prune`) use `networkx`, which is always installed as a core dependency.

Diagram **visualization** requires optional dependencies:

```bash
pip install matplotlib pygraphviz
```

If visualization dependencies are missing, `dj.Diagram` displays a warning and provides a stub class. Operational methods remain available regardless.
Diagram **visualization** additionally requires:

- **Graphviz** — the `dot` executable. The default notebook display, `_repr_svg_()`, goes through
`make_svg()` → `make_dot()` → pydot → `dot`, so this is the path nearly all usage hits. `pydot`
ships with DataJoint, but Graphviz itself is a system package that `pip` cannot install
(`brew install graphviz`, `sudo apt-get install graphviz`). Without it, rendering raises
`FileNotFoundError` — see
[Installation → Troubleshooting](../../how-to/installation.md#djdiagram-raises-filenotfounderror).
- **matplotlib** — only for `Diagram.draw()`, a separate `make_image()` path that most usage never
touches: `pip install datajoint[viz]`.

Operational methods remain available regardless.

---

Expand Down
93 changes: 8 additions & 85 deletions src/tutorials/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,76 +2,15 @@

Learn DataJoint by building real pipelines.

These tutorials guide you through building data pipelines step by step. Each tutorial
is a Jupyter notebook that you can run interactively. Start with the basics and
progress to domain-specific and advanced topics.
These tutorials guide you through building data pipelines step by step. Each tutorial is a
Jupyter notebook, published here with its code and its executed outputs, so you can follow the
whole pipeline without leaving the page. Start with the basics and progress to domain-specific
and advanced topics.

## Quick Start

Install DataJoint:

```bash
pip install datajoint
```

Configure database credentials in your project (see [Configuration](../reference/configuration.md)):

```bash
# Create datajoint.json for non-sensitive settings
echo '{"database": {"host": "localhost", "port": 3306}}' > datajoint.json

# Create secrets directory for credentials
mkdir -p .secrets
echo "root" > .secrets/database.user
echo "password" > .secrets/database.password
```

Define and populate a simple pipeline:

```python
import datajoint as dj

schema = dj.Schema('my_pipeline')

@schema
class Subject(dj.Manual):
definition = """
subject_id : int32
---
name : varchar(100)
date_of_birth : date
"""

@schema
class Session(dj.Manual):
definition = """
-> Subject
session_idx : int16
---
session_date : date
"""

@schema
class SessionAnalysis(dj.Computed):
definition = """
-> Session
---
result : float64
"""

def make(self, key):
# Compute result for this session
self.insert1({**key, 'result': 42.0})

# Insert data
Subject.insert1({'subject_id': 1, 'name': 'M001', 'date_of_birth': '2026-01-15'})
Session.insert1({'subject_id': 1, 'session_idx': 1, 'session_date': '2026-01-06'})

# Run computations
SessionAnalysis.populate()
```

Continue learning with the structured tutorials below.
**Want to run them yourself?** Every notebook can be downloaded and executed against your own
MySQL or PostgreSQL database. [Installation](../how-to/installation.md) covers the setup:
DataJoint, a database, Jupyter, and
[getting the notebooks](../how-to/installation.md#running-the-tutorial-notebooks).

## Learning Paths

Expand Down Expand Up @@ -198,19 +137,3 @@ Extending DataJoint for specialized use cases:
- [JSON Data Type](advanced/json-type.ipynb) — Semi-structured data in tables
- [Distributed Computing](advanced/distributed.ipynb) — Multi-process and cluster workflows
- [Custom Codecs](advanced/custom-codecs.ipynb) — Extending the type system

## Running the Tutorials

```bash
# Clone the repository
git clone https://github.com/datajoint/datajoint-docs.git
cd datajoint-docs

# Start the tutorial environment
docker compose up -d

# Launch Jupyter
jupyter lab src/tutorials/
```

All tutorials use a local MySQL database that resets between sessions.
Loading